diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index f9e7114..91b9f8d 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -3,7 +3,7 @@
repos:
- repo: https://github.com/DavidAnson/markdownlint-cli2
- rev: v0.9.2
+ rev: v0.13.0
hooks:
- id: markdownlint-cli2
language_version: "18.17.1"
@@ -11,7 +11,7 @@ repos:
rev: v2.2.5
hooks:
- id: codespell
- args: [--ignore-words=.codespellignore]
+ args: [--ignore-words=.codespellignore, --exclude-file=docs/example.ipynb]
types_or: [jupyter, markdown, python, shell]
- repo: https://github.com/psf/black
rev: 23.9.1
diff --git a/README.md b/README.md
index 9f95823..fe75d20 100644
--- a/README.md
+++ b/README.md
@@ -18,12 +18,99 @@
- [Browse the example in human-readable form](https://radiantearth.github.io/stac-browser/#/external/raw.githubusercontent.com/stactools-packages/noaa-hrrr/main/examples/collection.json)
- [Browse a notebook demonstrating the example item and collection](https://github.com/stactools-packages/noaa-hrrr/tree/main/docs/example.ipynb)
-A short description of the package and its usage.
+
+
+This package can be used to generate STAC metadata for the NOAA High Resolution Rapid
+Refresh (HRRR) atmospheric forecast dataset.
+
+The data are uploaded to cloud storage in AWS, Azure, and Google so you can pick
+which cloud provider you want to use for the `grib` and `index` hrefs using the
+`cloud_provider` argument to the functions in `stactools.noaa_hrrr.stac`.
+
+## Background
+
+The [NOAA HRRR dataset](https://www.nco.ncep.noaa.gov/pmb/products/hrrr/#CO))
+is a continuously updated atmospheric forecast data product.
+
+### Data structure
+
+- There are two regions: CONUS and Alaska
+- Every hour, new hourly forecasts are generated for many atmospheric attributes
+ for each region
+ - All hours (`00-23`) get an 18-hour forecast in the `conus` region
+ - Forecasts are generated every three hours (`00`, `03`, `06`, etc) in the
+ `alaska` region
+ - On hours `00`, `06`, `12`, `18` a 48-hour forecast is generated
+ - One of the products (`subh`) gets 15 minute forecasts (four per hour per
+ attribute), but the sub-hourly forecasts are stored as layers within a
+ single GRIB2 file for the forecast hour rather than in separate files.
+- The forecasts are broken up into 4 products (`sfc`, `prs`, `nat`, `subh`),
+- Each GRIB2 file has hundreds to thousands of variables
+- Each .grib2 file is accompanied by a .grib2.idx which has variable-level
+ metadata including the starting byte for the data in that variable (useful
+ for making range requests instead of reading the entire file) and some
+ other descriptive metadata
+
+### Summary of Considerations for Organizing STAC Metadata
+
+After extensive discussions, we decided to organize the STAC metadata with
+the following structure:
+
+1. **Collections**: Separate collections for each region-product combination
+ - regions: `conus` and `alaska`
+ - products: `sfc`, `prs`, `nat`, and `subh`
+
+2. **Items**: Each GRIB file in the archive is represented as an item with two assets:
+ - `"grib"`: Contains the actual data.
+ - `"index"`: The .grib2.idx sidecar file.
+
+ Each GRIB file contains the forecasts for all of a product's variables for a
+ particular forecast hour from a reference time, so you need to combine data
+ from multiple items to construct a time series for a forecast.
+
+3. **`grib:layers`**: Within each `"grib"` asset, a `grib:layers` property details
+ each layer's information, including description, units, and byte ranges.
+ This enables applications to access specific parts of the GRIB2 files without
+ downloading the entire file.
+
+ - We intend to propose a `GRIB` STAC extension with the `grib:layers` property
+ for storing byte-ranges after testing this specification out on other GRIB2
+ datasets.
+ - The layer-level metadata is worth storing in STAC because you can construct
+ URIs for specific layers that GDAL can read using either `/vsisubfile` or
+ `vrt://`:
+ - `/vsisubfile/{start_byte}_{byte_size},/vsicurl/{grib_href}`
+ - `vrt:///vsicurl/{grib_href}?bands={grib_message}`, where `grib_message` is
+ the index of the layer within the GRIB2 file.
+ - under the hood, GDAL's `vrt` driver is reading the sidecar .grib2.idx file
+ and translating it into a `/vsisubfile` URI.
+
+### Advantages
+
+- Applications can use `grib:layers` to create layer-specific data sets, facilitating
+efficient data handling.
+- Splitting by region and product allows defining coherent collection-level datacube
+metadata, enhancing accessibility.
+
+### Disadvantages
+
+- Storing layer-level metadata like byte ranges in the STAC metadata bloats the STAC
+ items because there are hundreds to thousands of layers in each GRIB2 file.
+
+For more details, please refer to the related [issue discussion](https://github.com/developmentseed/noaa-hrrr/issues/1)
+and pull requests [#3](https://github.com/developmentseed/noaa-hrrr/pull/3) and
+[#6](https://github.com/developmentseed/noaa-hrrr/pull/6).
## STAC examples
- [Collection](examples/collection.json)
-- [Item](examples/item/item.json)
+- [Item](examples/hrrr-conus-sfc-2024-05-10T12-FH0/hrrr-conus-sfc-2024-05-10T12-FH0.json)
+
+## Python usage example
+
+- Check out the [example notebook](./docs/example.ipynb) for examples of how to
+ create STAC metadata and how to use STAC items with `grib:layers` metadata to
+ load the data into xarray.
## Installation
diff --git a/docs/example.ipynb b/docs/example.ipynb
index 995d913..b005f1b 100644
--- a/docs/example.ipynb
+++ b/docs/example.ipynb
@@ -25,6 +25,101 @@
"cell_type": "code",
"execution_count": 1,
"metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime, timedelta\n",
+ "from pathlib import Path\n",
+ "from pystac import Collection, Item\n",
+ "from IPython.display import display\n",
+ "\n",
+ "import stactools.noaa_hrrr.stac\n",
+ "from stactools.noaa_hrrr.inventory import load_idx, read_idx\n",
+ "from stactools.noaa_hrrr.metadata import CloudProvider, Product, Region, parse_href\n",
+ "\n",
+ "\n",
+ "root = Path().cwd().parent\n",
+ "collection = Collection.from_file(str(root / \"examples\" / \"collection.json\"))\n",
+ "item = next(collection.get_items())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Creating items and collections\n",
+ "\n",
+ "Items can be created from a `region`, `product`, `cloud_provider`, `reference_datetime`, and `forecast_hour`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "item = stactools.noaa_hrrr.stac.create_item(\n",
+ " region=Region.conus,\n",
+ " product=Product.sfc,\n",
+ " cloud_provider=CloudProvider.azure,\n",
+ " reference_datetime=datetime.now() - timedelta(days=7),\n",
+ " forecast_hour=6,\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The `parse_href` function can be used to parse the argument values from an `href`:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "href = \"https://noaahrrr.blob.core.windows.net/hrrr/hrrr.20240613/conus/hrrr.t11z.wrfsfcf06.grib2\"\n",
+ "item = stactools.noaa_hrrr.create_item(**parse_href(href))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The `create_item` function will load the `.grib2.idx` sidecar file from cloud storage in order to populate the asset-level metadata.\n",
+ "If you already have the `.grib2.idx` file available in memory, you can use the `create_item_from_idx_df` function:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "idx_df = read_idx(href + \".idx\")\n",
+ "\n",
+ "item = stactools.noaa_hrrr.stac.create_item_from_idx_df(\n",
+ " idx_df=idx_df,\n",
+ " **parse_href(href)\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Items are added to collections using the standard pystac interface."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "scrolled": true
+ },
"outputs": [
{
"data": {
@@ -68,132848 +163,28 @@
" \n",
" \n",
"
\n",
- " description\n",
- " \"The NOAA HRRR is a real-time 3km resolution, hourly updated, cloud-resolving, convection-allowing atmospheric model, initialized by 3km grids with 3km radar assimilation. Radar data is assimilated in the HRRR every 15 min over a 1-hour period adding further detail to that provided by the hourly data assimilation from the 13km radar-enhanced Rapid Refresh (RAP) system. This specific collection represents 2D surface levels for the conus region.\"\n",
- "
\n",
- " description\n",
- " \"2D Surface Level forecast data as a grib2 file. Subsets of the data can be loaded using the provided byte range.\"\n",
- "
\n",
- " description\n",
- " \"The atmospheric level for which the forecast is applicable, e.g. surface, top of atmosphere, 100 m above ground, etc.\"\n",
- "
\n",
- " description\n",
- " \"2D Surface Level forecast data as a grib2 file. Subsets of the data can be loaded using the provided byte range.\"\n",
- "
\n",
- " description\n",
- " \"2D Surface Level forecast data as a grib2 file. Subsets of the data can be loaded using the provided byte range.\"\n",
- "
\n",
- " description\n",
- " \"2D Surface Level forecast data as a grib2 file. Subsets of the data can be loaded using the provided byte range.\"\n",
- "
\n",
- " description\n",
- " \"2D Surface Level forecast data as a grib2 file. Subsets of the data can be loaded using the provided byte range.\"\n",
- "
\n",
- " description\n",
- " \"The NOAA HRRR is a real-time 3km resolution, hourly updated, cloud-resolving, convection-allowing atmospheric model, initialized by 3km grids with 3km radar assimilation. Radar data is assimilated in the HRRR every 15 min over a 1-hour period adding further detail to that provided by the hourly data assimilation from the 13km radar-enhanced Rapid Refresh (RAP) system. This specific collection represents 2D surface levels for the conus region.\"\n",
- "
\n",
- " description\n",
- " \"2D Surface Level forecast data as a grib2 file. Subsets of the data can be loaded using the provided byte range.\"\n",
- "
\n",
- " description\n",
- " \"The atmospheric level for which the forecast is applicable, e.g. surface, top of atmosphere, 100 m above ground, etc.\"\n",
- "
PROJCS["unnamed",GEOGCS["Coordinate System imported from GRIB file",DATUM["unnamed",SPHEROID["Sphere",6371229,0]],PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]]],PROJECTION["Lambert_Conformal_Conic_2SP"],PARAMETER["latitude_of_origin",38.5],PARAMETER["central_meridian",-97.5],PARAMETER["standard_parallel_1",38.5],PARAMETER["standard_parallel_2",38.5],PARAMETER["false_easting",0],PARAMETER["false_northing",0],UNIT["Metre",1],AXIS["Easting",EAST],AXIS["Northing",NORTH]]
semi_major_axis :
6371229.0
semi_minor_axis :
6371229.0
inverse_flattening :
0.0
reference_ellipsoid_name :
Sphere
longitude_of_prime_meridian :
0.0
prime_meridian_name :
Greenwich
geographic_crs_name :
Coordinate System imported from GRIB file
horizontal_datum_name :
unnamed
projected_crs_name :
unnamed
grid_mapping_name :
lambert_conformal_conic
standard_parallel :
(38.5, 38.5)
latitude_of_projection_origin :
38.5
longitude_of_central_meridian :
-97.5
false_easting :
0.0
false_northing :
0.0
spatial_ref :
PROJCS["unnamed",GEOGCS["Coordinate System imported from GRIB file",DATUM["unnamed",SPHEROID["Sphere",6371229,0]],PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]]],PROJECTION["Lambert_Conformal_Conic_2SP"],PARAMETER["latitude_of_origin",38.5],PARAMETER["central_meridian",-97.5],PARAMETER["standard_parallel_1",38.5],PARAMETER["standard_parallel_2",38.5],PARAMETER["false_easting",0],PARAMETER["false_northing",0],UNIT["Metre",1],AXIS["Easting",EAST],AXIS["Northing",NORTH]]