A Comprehensive Python Toolkit for Accessing High‐Throughput Computing to Support Large Hydrologic Modeling Tasks

Keywords

high-throughput computing, computational methods, decision support systems, simulation, Python, cloud computing

Abstract

The National Flood Interoperability Experiment (NFIE) was an undertaking that initiated a transformation in national hydrologic forecasting by providing streamflow forecasts at high spatial resolution over the whole country. This type of large‐scale, high‐resolution hydrologic modeling requires flexible and scalable tools to handle the resulting computational loads. While high‐throughput computing (HTC) and cloud computing provide an ideal resource for large‐scale modeling because they are cost‐effective and highly scalable, nevertheless, using these tools requires specialized training that is not always common for hydrologists and engineers. In an effort to facilitate the use of HTC resources the National Science Foundation (NSF) funded project, CI‐WATER, has developed a set of Python tools that can automate the tasks of provisioning and configuring an HTC environment in the cloud, and creating and submitting jobs to that environment. These tools are packaged into two Python libraries: CondorPy and TethysCluster. Together these libraries provide a comprehensive toolkit for accessing HTC to support hydrologic modeling. Two use cases are described to demonstrate the use of the toolkit, including a web app that was used to support the NFIE national‐scale modeling.

Original Publication Citation

Christensen, Scott D., Nathan R. Swain, Norman L. Jones, E. James Nelson, Alan D. Snow, and Herman G. Dolder. "A Comprehensive Python Toolkit for Accessing High-Throughput Computing to Support Large Hydrologic Modeling Tasks."JAWRA Journal of the American Water Resources Association53, no. 2 (2017): 333-343.

Document Type

Peer-Reviewed Article

Publication Date

2016-09-13

Permanent URL

http://hdl.lib.byu.edu/1877/7079

Publisher

Journal of the American Water Resources Association

Language

English

College

Ira A. Fulton College of Engineering and Technology

Department

Civil and Environmental Engineering

University Standing at Time of Publication

Full Professor

Link to Full Text

Share

COinS