Why Jupyter is data scientists’ computational notebook of choice

Jeffrey M. Perkel

DOI: https://doi.org/10.1038/d41586-018-07196-1

IF: 64.8

2018-10-30

Nature

Abstract:An improved architecture and enthusiastic user base are driving uptake of the open-source web tool.

multidisciplinary sciences

What problem does this paper attempt to address?

What's Wrong with Computational Notebooks? Pain Points, Needs, and Design Opportunities

Souti Chattopadhyay,Ishita Prasad,Austin Z. Henley,Anita Sarma,Titus Barik

DOI: https://doi.org/10.1145/3313831.3376729

2020-04-21

Abstract:Computational notebooks - such as Azure, Databricks, and Jupyter - are a popular, interactive paradigm for data scientists to author code, analyze data, and interleave visualizations, all within a single document. Nevertheless, as data scientists incorporate more of their activities into notebooks, they encounter unexpected difficulties, or pain points, that impact their productivity and disrupt their workflow. Through a systematic, mixed-methods study using semi-structured interviews (n=20) and survey (n=156) with data scientists, we catalog nine pain points when working with notebooks. Our findings suggest that data scientists face numerous pain points throughout the entire workflow - from setting up notebooks to deploying to production - across many notebook environments. Our data scientists report essential notebook requirements, such as supporting data exploration and visualization. The results of our study inform and inspire the design of computational notebooks.
Using Jupyter for Reproducible Scientific Workflows

Marijan Beg,Juliette Taka,Thomas Kluyver,Alexander Konovalov,Min Ragan-Kelley,Nicolas M. Thiery,Hans Fangohr

DOI: https://doi.org/10.1109/mcse.2021.3052101

2021-03-01

Abstract:Literate computing has emerged as an important tool for computational studies and open science, with growing folklore of best practices. In this work, we report two case studies—one in computational magnetism and another in computational mathematics—where domain-specific software was exposed to the Jupyter environment. This enables high level control of simulations and computation, interactive exploration of computational results, batch processing on HPC resources, and reproducible workflow documentation in Jupyter notebooks. In the first study, Ubermag drives existing computational micromagnetics software through a domain-specific language embedded in Python. In the second study, a dedicated Jupyter kernel interfaces with the GAP system for computational discrete algebra and its dedicated programming language. In light of these case studies, we discuss the benefits of this approach, including progress toward more reproducible and reusable research results and outputs, notably through the use of infrastructure such as JupyterHub and Binder.

computer science, interdisciplinary applications
Data Makes Better Data Scientists

Jinjin Zhao,Avidgor Gal,Sanjay Krishnan

2024-05-28

Abstract:With the goal of identifying common practices in data science projects, this paper proposes a framework for logging and understanding incremental code executions in Jupyter notebooks. This framework aims to allow reasoning about how insights are generated in data science and extract key observations into best data science practices in the wild. In this paper, we show an early prototype of this framework and ran an experiment to log a machine learning project for 25 undergraduate students.

Human-Computer Interaction
Jupyter-Enabled Astrophysical Analysis for Researchers and Students

Stéphanie Juneau,Knut A. G. Olsen,Robert Nikutta,Alice Jacques,Stephen Bailey

DOI: https://doi.org/10.1109/MCSE.2021.3057097

2021-04-14

Abstract:The advent of increasingly large and complex datasets has fundamentally altered the way that scientists conduct astronomy research. The need to work closely to the data has motivated the creation of online science platforms, which include a suite of software tools and services, therefore going beyond data storage and data access. We present two example applications of Jupyter as a part of astrophysical science platforms for professional researchers and students. First, the Astro Data Lab is developed and operated by NOIRLab with a mission to serve the astronomy community with now over 1500 registered users. Second, the Dark Energy Spectroscopic Instrument science platform serves its geographically distributed team comprising about 900 collaborators from over 90 institutions. We describe the main uses of Jupyter and the interfaces that needed to be created to embed it within science platform ecosystems. We use these examples to illustrate the broader concept of empowering researchers and providing them with access to not only large datasets but also cutting-edge software, tools, and data services without requiring any local installation, which can be relevant for a wide range of disciplines. Future advances may involve science platform networks, and tools for simultaneously developing Jupyter notebooks to facilitate collaborations.

Instrumentation and Methods for Astrophysics,Astrophysics of Galaxies
Better Code, Better Sharing:On the Need of Analyzing Jupyter Notebooks

Jiawei Wang,Li Li,Andreas Zeller

DOI: https://doi.org/10.48550/arXiv.1906.05234

2019-06-13

Abstract:By bringing together code, text, and examples, Jupyter notebooks have become one of the most popular means to produce scientific results in a productive and reproducible way. As many of the notebook authors are experts in their scientific fields, but laymen with respect to software engineering, one may ask questions on the quality of notebooks and their code. In a preliminary study, we experimentally demonstrate that Jupyter notebooks are inundated with poor quality code, e.g., not respecting recommended coding practices, or containing unused variables and deprecated functions. Considering the education nature of Jupyter notebooks, these poor coding practices as well as the lacks of quality control might be propagated into the next generation of developers. Hence, we argue that there is a strong need to programmatically analyze Jupyter notebooks, calling on our community to pay more attention to the reliability of Jupyter notebooks.

Software Engineering
Better Code, Better Sharing: On the Need of Analyzing Jupyter Notebooks

Jiawei Wang,Li Li,Andreas Zeller

DOI: https://doi.org/10.1145/3377816.3381724

2020-01-01

Abstract:By bringing together code, text, and examples, Jupyter notebooks have become one of the most popular means to produce scientific results in a productive and reproducible way. As many of the notebook authors are experts in their scientific fields, but laymen with respect to software engineering, one may ask questions on the quality of notebooks and their code. In a preliminary study, we experimentally demonstrate that Jupyter notebooks are inundated with poor quality code, e.g., not respecting recommended coding practices, or containing unused variables and deprecated functions. Considering the education nature of Jupyter notebooks, these poor coding practices, as well as the lacks of quality control, might be propagated into the next generation of developers. Hence, we argue that there is a strong need to programmatically analyze Jupyter notebooks, calling on our community to pay more attention to the reliability of Jupyter notebooks.
Better Code, Better Sharing

Jiawei Wang,Li,Andreas Zeller

DOI: https://doi.org/10.1145/3377816.3381724

2020-01-01

Abstract:By bringing together code, text, and examples, Jupyter notebooks have become one of the most popular means to produce scientific results in a productive and reproducible way. As many of the notebook authors are experts in their scientific fields, but laymen with respect to software engineering, one may ask questions on the quality of notebooks and their code. In a preliminary study, we experimentally demonstrate that Jupyter notebooks are inundated with poor quality code, e.g., not respecting recommended coding practices, or containing unused variables and deprecated functions. Considering the education nature of Jupyter notebooks, these poor coding practices, as well as the lacks of quality control, might be propagated into the next generation of developers. Hence, we argue that there is a strong need to programmatically analyze Jupyter notebooks, calling on our community to pay more attention to the reliability of Jupyter notebooks.
A College Teacher’s Introduction to Jupyter Notebooks

Paul Isihara

DOI: https://doi.org/10.1564/tme_v28.4.03

2021-12-01

Abstract:This introduction to Jupyter Notebooks (JNBs) is an outcome of a year of learning, creating, and classroom testing JNBs in a variety of settings: a sabbatical with an urban violence reduction organization; an after school distance learning program for low to middle income high school students in Chicago; a general education computer-lab based math course; and finally two upper division courses for applied math majors at a liberal arts college. The examples include line graphs, histograms, and a pie chart of violent crime data created for a community organization, a high school/general education lab on computer music, a visualization of volumes computable by triple integrals, and creation of a .gif animation of a numerical solution to a 2-Dimensional PDE. In addition to executing and displaying output of computer code, JNBs can import data from files and external websites, interpret equations written in LaTeX, display PowerPoints in pdf format, and show videos posted on other websites. JNBs can also save output data, figures, and animations to files in a variety of formats. At a general education level, JNBs can enrich student learning experience by helping students overcome math anxiety through inclusion of the arts. At the undergraduate math major level, JNBs can create interesting visualizations/animations and support service learning projects. The appendix explains how to get started using JNBs. The learning curve for creating the JNBs described in this article is not that steep, and the low-hanging fruit from using JNBs for a wide-range of teaching contexts and for community service learning projects is plentiful.
Interactive Lab Notebooks for Robotics Researchers

Rolando Garcia

2024-05-14

Abstract:Interactive notebooks, such as Jupyter, have revolutionized the field of data science by providing an integrated environment for data, code, and documentation. However, their adoption by robotics researchers and model developers has been limited. This study investigates the logging and record-keeping practices of robotics researchers, drawing parallels to the pre-interactive notebook era of data science. Through interviews with robotics researchers, we identified the reliance on diverse and often incompatible tools for managing experimental data, leading to challenges in reproducibility and data traceability. Our findings reveal that robotics researchers can benefit from a specialized version of interactive notebooks that supports comprehensive data entry, continuous context capture, and agile data staging. We propose extending interactive notebooks to better serve the needs of robotics researchers by integrating features akin to traditional lab notebooks. This adaptation aims to enhance the organization, analysis, and reproducibility of experimental data in robotics, fostering a more streamlined and efficient research workflow.

Computational Engineering, Finance, and Science
Wrex: A Unified Programming-by-Example Interaction for Synthesizing Readable Code for Data Scientists

Ian Drosos,Titus Barik,Philip J. Guo,Robert DeLine,Sumit Gulwani

DOI: https://doi.org/10.1145/3313831.3376442

2020-04-21

Abstract:Data wrangling is a difficult and time-consuming activity in computational notebooks, and existing wrangling tools do not fit the exploratory workflow for data scientists in these environments. We propose a unified interaction model based on programming-by-example that generates readable code for a variety of useful data transformations, implemented as a Jupyter notebook extension called Wrex. User study results demonstrate that data scientists are significantly more effective and efficient at data wrangling with Wrex over manual programming. Qualitative participant feedback indicates that Wrex was useful and reduced barriers in having to recall or look up the usage of various data transform functions. The synthesized code allowed data scientists to verify the intended data transformation, increased their trust and confidence in Wrex, and fit seamlessly within their cell-based notebook workflows. This work suggests that presenting readable code to professional data scientists is an indispensable component of offering data wrangling tools in notebooks.
WHATSNEXT: Guidance-enriched Exploratory Data Analysis with Interactive, Low-Code Notebooks

Chen Chen,Jane Hoffswell,Shunan Guo,Ryan Rossi,Yeuk-Yin Chan,Fan Du,Eunyee Koh,Zhicheng Liu

2023-08-19

Abstract:Computational notebooks such as Jupyter are popular for exploratory data analysis and insight finding. Despite the module-based structure, notebooks visually appear as a single thread of interleaved cells containing text, code, visualizations, and tables, which can be unorganized and obscure users' data analysis workflow. Furthermore, users with limited coding expertise may struggle to quickly engage in the analysis process. In this work, we design and implement an interactive notebook framework, WHATSNEXT, with the goal of supporting low-code visual data exploration with insight-based user guidance. In particular, we (1) re-design a standard notebook cell to include a recommendation panel that suggests possible next-step exploration questions or analysis actions to take, and (2) create an interactive, dynamic tree visualization that reflects the analytic dependencies between notebook cells to make it easy for users to see the structure of the data exploration threads and trace back to previous steps.

Human-Computer Interaction
Pynblint: A quality assurance tool to improve the quality of Python Jupyter notebooks

Luigi Quaranta,Fabio Calefato,Filippo Lanubile

DOI: https://doi.org/10.1016/j.softx.2024.101959

IF: 2.868

2024-11-18

SoftwareX

Abstract:Jupyter Notebook is widely recognized as a crucial tool for data science professionals and students. Its interactive and self-documenting nature makes it particularly suitable for data-driven programming tasks. Nonetheless, it faces criticism for its limited support for software engineering best practices and its tendency to encourage bad programming habits, such as non-linear code execution. These issues often result in non-reproducible, poorly documented, and low-quality notebook code. In this paper, we introduce Pynblint , a static analyzer for Python Jupyter notebooks. Pynblint is designed to help data scientists write better notebooks, easy to understand and reproduce. We report on how we validated Pynblint with both professional data scientists and students, receiving overall positive feedback. Additionally, we discuss the potential of Pynblint to facilitate research inquiries into computational notebooks.

computer science, software engineering
Bug Analysis in Jupyter Notebook Projects: An Empirical Study

Taijara Loiola de Santana,Paulo Anselmo da Mota Silveira Neto,Eduardo Santana de Almeida,Iftekhar Ahmed

DOI: https://doi.org/10.1145/3641539

IF: 3.685

2024-01-22

ACM Transactions on Software Engineering and Methodology

Abstract:Computational notebooks, such as Jupyter, have been widely adopted by data scientists to write code for analyzing and visualizing data. Despite their growing adoption and popularity, few studies were found to understand Jupyter development challenges from the practitioners’ point of view. This paper presents a systematic study of bugs and challenges that Jupyter practitioners face through a large-scale empirical investigation. We mined 14,740 commits from 105 GitHub open-source projects with Jupyter notebook code. Next, we analyzed 30,416 Stack Overflow posts, which gave us insights into bugs that practitioners face when developing Jupyter notebook projects. Next, we conducted nineteen interviews with data scientists to uncover more details about Jupyter bugs and to gain insight into Jupyter developers’ challenges. Finally, to validate the study results and proposed taxonomy, we conducted a survey with 91 data scientists. We also highlight bug categories, their root causes, and the challenges that Jupyter practitioners face.

computer science, software engineering
Enhancing Comprehension and Navigation in Jupyter Notebooks with Static Analysis

Ashwin Prasad Shivarpatna Venkatesh,Jiawei Wang,Li,Eric Bodden

DOI: https://doi.org/10.1109/saner56733.2023.00044

2023-01-01

Abstract:Jupyter notebooks enable developers to interleave code snippets with rich-text and in-line visualizations. Data scientists use Jupyter notebook as the de-facto standard for creating and sharing machine-learning based solutions, primarily written in Python. Recent studies have demonstrated, however, that a large portion of Jupyter notebooks available on public platforms are undocumented and lacks a narrative structure. This reduces the readability of these notebooks. To address this shortcoming, this paper presents HeaderGen, a novel toolbased approach that automatically annotates code cells with categorical markdown headers based on a taxonomy of machinelearning operations, and classifies and displays function calls according to this taxonomy. For this functionality to be realized, HeaderGen enhances an existing call graph analysis in PyCG. To improve precision, HeaderGen extends PyCG's analysis with support for handling external library code and flow-sensitivity. The former is realized by facilitating the resolution of function return-types. Furthermore, HeaderGen uses type information to perform pattern matching on code syntax to annotate code cells. The evaluation on 15 real-world Jupyter notebooks from Kaggle shows that HeaderGen's underlying call graph analysis yields high accuracy (96.4% precision and 95.9% recall). This is because HeaderGen can resolve return-types of external libraries where existing type inference tools such as pytype (by Google), pyright (by Microsoft), and Jedi fall short. The header generation has a precision of 82.2% and a recall rate of 96.8% with regard to headers created manually by experts. In a user study, HeaderGen helps participants finish comprehension and navigation tasks faster. All participants clearly perceive HeaderGen as useful to their task.
GIS and Computational Notebooks

Geoff Boeing,Dani Arribas-Bel

DOI: https://doi.org/10.48550/arXiv.2101.00351

2021-01-02

Abstract:Researchers and practitioners across many disciplines have recently adopted computational notebooks to develop, document, and share their scientific workflows - and the GIS community is no exception. This chapter introduces computational notebooks in the geographical context. It begins by explaining the computational paradigm and philosophy that underlie notebooks. Next it unpacks their architecture to illustrate a notebook user's typical workflow. Then it discusses the main benefits notebooks offer GIS researchers and practitioners, including better integration with modern software, more natural access to new forms of data, and better alignment with the principles and benefits of open science. In this context, it identifies notebooks as the glue that binds together a broader ecosystem of open source packages and transferable platforms for computational geography. The chapter concludes with a brief illustration of using notebooks for a set of basic GIS operations. Compared to traditional desktop GIS, notebooks can make spatial analysis more nimble, extensible, and reproducible and have thus evolved into an important component of the geospatial science toolkit.

Computers and Society,Computation
Teachers' perception of Jupyter and R Shiny as digital tools for open education and science

Jozef Hanč,Peter Štrauch,Eva Paňková,Martina Hančová

DOI: https://doi.org/10.48550/arXiv.2007.11262

2020-07-24

Abstract:During the last ten years advances in open-source digital technology, used especially by data science, led to very accessible ways how to obtain, store, process, analyze or share data in almost every human activity. Data science tools bring not only transparency, accessibility, and reproducibility in open science, but also give benefits in open education as learning tools for improving effectiveness of instruction. Together with our pedagogical introduction and review of Jupyter as an interactive multimedia learning tool, we present our three-years long research in the framework of a complex mixed-methods approach which examines physics teachers' perception of Jupyter technology in three groups: Ph.D. candidates in physics education research (PER) ($N = 9$), pre-service physics teachers ($N = 33$) and in-service physics teachers ($N = 40$). Despite the fact that open-source Jupyter notebooks are natural and easy as email or web, the results suggest that in-service teachers are not prepared for Jupyter technology and open analysis, but positively accept open education data presented via another open-source data science tool, R Shiny interactive web application, as an important form of immediate feedback and learning about the quality of their instruction. Simultaneously our instruction results in the frame of the Flipped Learning also indicate that young beginning PER researchers and pre-service physics teachers can master key digital skills to work with Jupyter technology appreciating its big impact on their learning, data and statistical literacy or professional development. All results support the ongoing worldwide effort to implement Jupyter in traditional education as a promising free open-source interactive learning tool to foster learning process, especially for the upcoming young generation.

Physics Education
You Only Write Thrice: Creating Documents, Computational Notebooks and Presentations From a Single Source

Kacper Sokol,Peter Flach

DOI: https://doi.org/10.48550/arXiv.2107.06639

2021-07-02

Programming Languages

Abstract:Academic trade requires juggling multiple variants of the same content published in different formats: manuscripts, presentations, posters and computational notebooks. The need to track versions to accommodate for the write--review--rebut--revise life-cycle adds another layer of complexity. We propose to significantly reduce this burden by maintaining a single source document in a version-controlled environment (such as git), adding functionality to generate a collection of output formats popular in academia. To this end, we utilise various open-source tools from the Jupyter scientific computing ecosystem and operationalise selected software engineering concepts. We offer a proof-of-concept workflow that composes Jupyter Book (an online document), Jupyter Notebook (a computational narrative) and reveal.js slides from a single markdown source file. Hosted on GitHub, our approach supports change tracking and versioning, as well as a transparent review process based on the underlying code issue management infrastructure. An exhibit of our workflow can be previewed at https://so-cool.github.io/you-only-write-thrice/.
FAIR Jupyter: a knowledge graph approach to semantic sharing and granular exploration of a computational notebook reproducibility dataset

Sheeba Samuel,Daniel Mietchen

2024-04-19

Abstract:The way in which data are shared can affect their utility and reusability. Here, we demonstrate how data that we had previously shared in bulk can be mobilized further through a knowledge graph that allows for much more granular exploration and interrogation. The original dataset is about the computational reproducibility of GitHub-hosted Jupyter notebooks associated with biomedical publications. It contains rich metadata about the publications, associated GitHub repositories and Jupyter notebooks, and the notebooks' reproducibility. We took this dataset, converted it into semantic triples and loaded these into a triple store to create a knowledge graph, FAIR Jupyter, that we made accessible via a web service. This enables granular data exploration and analysis through queries that can be tailored to specific use cases. Such queries may provide details about any of the variables from the original dataset, highlight relationships between them or combine some of the graph's content with materials from corresponding external resources. We provide a collection of example queries addressing a range of use cases in research and education. We also outline how sets of such queries can be used to profile specific content types, either individually or by class. We conclude by discussing how such a semantically enhanced sharing of complex datasets can both enhance their FAIRness, i.e., their findability, accessibility, interoperability, and reusability, and help identify and communicate best practices, particularly with regards to data quality, standardization, automation and reproducibility.

Computational Engineering, Finance, and Science
Urban Street Network Analysis in a Computational Notebook

Geoff Boeing

DOI: https://doi.org/10.18335/region.v6i3.278

2020-01-18

Abstract:Computational notebooks offer researchers, practitioners, students, and educators the ability to interactively conduct analytics and disseminate reproducible workflows that weave together code, visuals, and narratives. This article explores the potential of computational notebooks in urban analytics and planning, demonstrating their utility through a case study of OSMnx and its tutorials repository. OSMnx is a Python package for working with OpenStreetMap data and modeling, analyzing, and visualizing street networks anywhere in the world. Its official demos and tutorials are distributed as open-source Jupyter notebooks on GitHub. This article showcases this resource by documenting the repository and demonstrating OSMnx interactively through a synoptic tutorial adapted from the repository. It illustrates how to download urban data and model street networks for various study sites, compute network indicators, visualize street centrality, calculate routes, and work with other spatial data such as building footprints and points of interest. Computational notebooks help introduce methods to new users and help researchers reach broader audiences interested in learning from, adapting, and remixing their work. Due to their utility and versatility, the ongoing adoption of computational notebooks in urban planning, analytics, and related geocomputation disciplines should continue into the future.

Physics and Society,Computation
Towards Transparent, Reusable, and Customizable Data Science in Computational Notebooks

Frederick Choi,Sajjadur Rahman,Hannah Kim,Dan Zhang

DOI: https://doi.org/10.1145/3544549.3585807

2023-03-24

Abstract:Data science workflows are human-centered processes involving on-demand programming and analysis. While programmable and interactive interfaces such as widgets embedded within computational notebooks are suitable for these workflows, they lack robust state management capabilities and do not support user-defined customization of the interactive components. The absence of such capabilities hinders workflow reusability and transparency while limiting the scope of exploration of the end-users. In response, we developed MAGNETON, a framework for authoring interactive widgets within computational notebooks that enables transparent, reusable, and customizable data science workflows. The framework enhances existing widgets to support fine-grained interaction history management, reusable states, and user-defined customizations. We conducted three case studies in a real-world knowledge graph construction and serving platform to evaluate the effectiveness of these widgets. Based on the observations, we discuss future implications of employing MAGNETON widgets for general-purpose data science workflows.

Human-Computer Interaction,Databases

Why Jupyter is data scientists’ computational notebook of choice

What's Wrong with Computational Notebooks? Pain Points, Needs, and Design Opportunities

Using Jupyter for Reproducible Scientific Workflows

Data Makes Better Data Scientists

Jupyter-Enabled Astrophysical Analysis for Researchers and Students

Better Code, Better Sharing:On the Need of Analyzing Jupyter Notebooks

Better Code, Better Sharing: On the Need of Analyzing Jupyter Notebooks

Better Code, Better Sharing

A College Teacher’s Introduction to Jupyter Notebooks

Interactive Lab Notebooks for Robotics Researchers

Wrex: A Unified Programming-by-Example Interaction for Synthesizing Readable Code for Data Scientists

WHATSNEXT: Guidance-enriched Exploratory Data Analysis with Interactive, Low-Code Notebooks

Pynblint: A quality assurance tool to improve the quality of Python Jupyter notebooks

Bug Analysis in Jupyter Notebook Projects: An Empirical Study

Enhancing Comprehension and Navigation in Jupyter Notebooks with Static Analysis

GIS and Computational Notebooks

Teachers' perception of Jupyter and R Shiny as digital tools for open education and science

You Only Write Thrice: Creating Documents, Computational Notebooks and Presentations From a Single Source

FAIR Jupyter: a knowledge graph approach to semantic sharing and granular exploration of a computational notebook reproducibility dataset

Urban Street Network Analysis in a Computational Notebook

Towards Transparent, Reusable, and Customizable Data Science in Computational Notebooks