Managing Big Data Output from the U.S. National Water Model

The National Water Model (NWM) is a newly designed national-scale hydrologic model that provides streamflow forecasts for the entire continental United States. Since released in August 2016, its daily outputs are about 400GB worth of NetCDF files, which has proven challenging in terms of data storage, mobility, and accessibility. As a complement to the 48-hour storage of model forecast data on the NOAA NOMADS server, the CUAHSI HydroShare project has been archiving the NWM outputs -- extending storage duration up to 40 days. We have added new capabilities to HydroShare to meet the requirement for long-term storage of regional NWM data to support research or applications such as replication and validation, cross-model comparison, and historical data analysis (e.g. hurricane or flooding events). We will present the design and implementation of a GIS-based subsetting tool package that enables users to subset NWM data for archival and analysis purposes. The subsetted data contains model outputs that geographically fall in a user-specified watershed polygon, and the resulting files follow the original NWM NetCDF file conventions being compatible with existing NWM tools. This tool package has been integrated by the latest NWM Forecast Viewer Tethys App which is part of the HydroShare web app environment. All of its key functions have been exposed through a simple user interface and also through machine accessible application programmer interfaces (APIs).