Design and Implementation of a Big Query Dataset and Application Programmer Interface (API) for the U.S. National Water Model

Kel N. Markert,Gui da Silva,Daniel P. Ames,Iman Maghami,Gustavious P. Williams,E. James Nelson,James Halgren,Arpita Patel,Adler Santos,Michael J. Ames
DOI: https://doi.org/10.1016/j.envsoft.2024.106123
IF: 5.471
2024-06-21
Environmental Modelling & Software
Abstract:We introduce an open-source web-based Application Programming Interface (API) developed within a representational state transfer (REST) architecture framework that provides access to the operational streamflow forecasts from the U.S. National Water Model (NWM). We built this API within the Google Cloud infrastructure, taking advantage of Google's API Gateway, BigQuery, and the Google Cloud Run architecture. We ran a data transformation from netCDF to tabular formation then ingested data into BigQuery using Apache Beam parallel processing technology. The API activates functions deployed to Cloud Run that executes SQL queries within the BigQuery data warehouse. The API gives users granular control to specify queries based on forecast type, reference datetime, stream segment specifics, and forecast ensemble members. This API greatly simplifies access and use of current and historic NWM forecasts, an otherwise arduous task due to the large volume of data and unwieldy storage methods. Retrieved forecast data can be used for many critical water management applications providing actionable intelligence for, e.g., flood management, safety, recreation, agriculture, and related uses.
environmental sciences,engineering, environmental,water resources,computer science, interdisciplinary applications
What problem does this paper attempt to address?