{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Fetching USGS and NWM Streamflow Data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Overview\n", "In this guide we'll demonstrate fetching National Water Model (NWM) streamflow forecasts from Google Cloud Storage (GCS). This example makes use of a pre-generated Evaluation dataset stored in TEEHR's examples data module. It contains a single USGS gage location and the corresponding NWM location ID.\n", "\n", "**Note**: For demonstration purposes several cells below are shown in markdown form. If you want to download this notebook and run them yourself, you will need to convert them to code cells.\n", "\n", "For a refresher on loading location and location crosswalk data into a new Evaluation refer back to the [](/user_guide/notebooks/02_loading_local_data.ipynb) and [](/user_guide/notebooks/04_setup_simple_example.ipynb) user guide pages." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Set up the example Evaluation" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from datetime import datetime\n", "from pathlib import Path\n", "import shutil\n", "\n", "import teehr\n", "from teehr.examples.setup_nwm_streamflow_example import setup_nwm_example\n", "\n", "# Tell Bokeh to output plots in the notebook\n", "from bokeh.io import output_notebook\n", "output_notebook()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Define the directory where the Evaluation will be created.\n", "test_eval_dir = Path(Path().home(), \"temp\", \"10_fetch_nwm_data\")\n", "shutil.rmtree(test_eval_dir, ignore_errors=True)\n", "\n", "# Setup the example evaluation using data from the TEEHR repository.\n", "setup_nwm_example(tmpdir=test_eval_dir)\n", "\n", "# Initialize the evaluation.\n", "ev = teehr.Evaluation(dir_path=test_eval_dir)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This example Evaluation only contains a single location, the USGS gage on the New River at Radford, VA." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "locations_gdf = ev.locations.to_geopandas()\n", "locations_gdf" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The ID of the National Water Model reach corresponding to the USGS gage is in the `location crosswalks` table." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "location_crosswalks_df = ev.location_crosswalks.to_pandas()\n", "location_crosswalks_df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Fetching USGS streamgage data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since the example Evalution already contains the location and location crosswalk IDs for USGS and NWM locations, we can make use of TEEHR's built-in tools to fetch USGS and NWM streamflow data.\n", "\n", "First we'll fetch USGS data from the National Water Information System (NWIS). TEEHR makes use of the USGS [dataretrieval](https://github.com/DOI-USGS/dataretrieval-python) python tool under the hood.\n", "\n", "Note that the USGS and NWM streamflow timeseries data have been pre-loading into this example Evaluation. However you can still download this notebook and execute the methods yourself." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`ev.fetch.usgs_streamflow()` is a method that fetches the USGS data for the primary locations in the evaluation. It requires users to define the start and end times of data to fetch and has several optional arguments. For more details on the method see: " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll fetch streamflow data at this gage during the 2024 Hurricane Helene event." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "# Convert this to a code cell to run locally\n", "ev.fetch.usgs_streamflow(\n", " start_date=datetime(2024, 9, 26),\n", " end_date=datetime(2024, 10, 1)\n", ")\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "TEEHR automatically loads the data into the Evaluation. USGS data is loaded a `primary` timeseries by default." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df = ev.primary_timeseries.to_pandas()\n", "df.teehr.timeseries_plot()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Fetching NWM streamflow data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`ev.fetch.nwm_operational_points()` is a method that fetches near real-time NWM point data (e.g., streamflow) from Google Cloud Storage. This method fetches data for the secondary location IDs listed in the `location_crosswalks` table, and automatically loads the time series into the `secondary_timeseries` Evaluation table.\n", "\n", "There are several required arguments to define when using the method, including the NWM configuration, NWM variable name, start date, number of ingest days, and others. Several optional arguments are also available." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll now fetch streamflow forecasts for the NWM location corresponding to the USGS gage." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ ":::{note}\n", "The tools for fetching NWM data in TEEHR can take advantage of `Dask`. Start a Dask cluster for improved performance when fetching NWM data if you have [Dask.Distributed installed](https://distributed.dask.org/en/latest/install.html)!\n", ":::" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "# Convert this to a code cell to run locally\n", "from dask.distributed import Client\n", "client = Client()\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "# Convert this to a code cell to run locally\n", "ev.fetch.nwm_operational_points(\n", " nwm_configuration=\"medium_range_mem1\",\n", " output_type=\"channel_rt_1\",\n", " variable_name=\"streamflow\",\n", " start_date=datetime(2024, 9, 26),\n", " ingest_days=1,\n", " nwm_version=\"nwm30\",\n", " add_configuraton_name=False\n", ")\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here we are fetching NWM version 3.0 Medium Range streamflow forecast, ensemble member 1, for the same time period as the USGS data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A list of available NWM configurations for point data is shown below. Appropriate values for the `output_type` and `variable_name` arguments depend on the specified `nwm_configuration` value.\n", "\n", "More information on NWM configurations can be found here: [https://water.noaa.gov/about/nwm](https://water.noaa.gov/about/nwm)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from teehr.models.fetching.nwm30_point import ConfigurationsEnum\n", "list(ConfigurationsEnum.__members__)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df = ev.secondary_timeseries.to_pandas()\n", "df.teehr.timeseries_plot()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now you can create the `joined_timeseries` table and calculate metrics comparing the NWM forecasts to the USGS observations." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ev.joined_timeseries.create()\n", "df = ev.joined_timeseries.to_pandas()\n", "df.head()" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "ev.spark.stop()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Additional NWM fetching methods in TEEHR" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### NWM Retrospective Point Data\n", "\n", "NWM retrospective point data for versions 2.0, 2.1, and 3.0:\n", "\n", "```{eval-rst}\n", ".. currentmodule:: teehr.evaluation.fetch\n", ".. autosummary::\n", " :recursive:\n", " :nosignatures:\n", "\n", " Fetch.nwm_retrospective_points\n", "```\n", "\n", "#### NWM Retrospective and Forecast Gridded Data\n", "\n", "NWM gridded data can also be fetched in TEEHR. Gridded data is summarized to zones (polygons) as the zonal mean. \n", "\n", "```{eval-rst}\n", ".. currentmodule:: teehr.evaluation.fetch\n", ".. autosummary::\n", " :recursive:\n", " :nosignatures:\n", "\n", " Fetch.nwm_operational_grids\n", " Fetch.nwm_retrospective_grids\n", "```\n", "\n", "Before gridded data can be fetched and summarized, the pixel weights (fractional area intersecting the zones) must be calculated.\n", "\n", "```{eval-rst}\n", ".. currentmodule:: teehr\n", ".. autosummary::\n", " :nosignatures:\n", " :recursive:\n", "\n", " teehr.utilities.generate_weights\n", "```" ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" } }, "nbformat": 4, "nbformat_minor": 2 }