A Comprehensive Python Toolkit for Accessing High‐Throughput Computing to Support Large Hydrologic Modeling Tasks
high-throughput computing, computational methods, decision support systems, simulation, Python, cloud computing
The National Flood Interoperability Experiment (NFIE) was an undertaking that initiated a transformation in national hydrologic forecasting by providing streamflow forecasts at high spatial resolution over the whole country. This type of large‐scale, high‐resolution hydrologic modeling requires flexible and scalable tools to handle the resulting computational loads. While high‐throughput computing (HTC) and cloud computing provide an ideal resource for large‐scale modeling because they are cost‐effective and highly scalable, nevertheless, using these tools requires specialized training that is not always common for hydrologists and engineers. In an effort to facilitate the use of HTC resources the National Science Foundation (NSF) funded project, CI‐WATER, has developed a set of Python tools that can automate the tasks of provisioning and configuring an HTC environment in the cloud, and creating and submitting jobs to that environment. These tools are packaged into two Python libraries: CondorPy and TethysCluster. Together these libraries provide a comprehensive toolkit for accessing HTC to support hydrologic modeling. Two use cases are described to demonstrate the use of the toolkit, including a web app that was used to support the NFIE national‐scale modeling.
Original Publication Citation
Christensen, Scott D., Nathan R. Swain, Norman L. Jones, E. James Nelson, Alan D. Snow, and Herman G. Dolder. "A Comprehensive Python Toolkit for Accessing High-Throughput Computing to Support Large Hydrologic Modeling Tasks."JAWRA Journal of the American Water Resources Association53, no. 2 (2017): 333-343.
BYU ScholarsArchive Citation
Christensen, Scott D.; Swain, Nathan R.; Jones, Norman L.; Nelson, E. James; Snow, Alan D.; and Dolder, Herman G., "A Comprehensive Python Toolkit for Accessing High‐Throughput Computing to Support Large Hydrologic Modeling Tasks" (2016). Faculty Publications. 4271.
Journal of the American Water Resources Association
Ira A. Fulton College of Engineering and Technology
Civil and Environmental Engineering
© 2016 American Water Resources Association
Copyright Use Information