feat: define separate collections for each product, set each GRIB layer as an asset #5
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Split into collections by region/product/forecast hour set
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
hrodmn
changed the title
hrodmn
changed the title
|
Closing in favor of #6 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is an approach to #1 that treats individual slices of the GRIB files as assets. It is a big change from the approach in #3 because it a) does not depend on
Herbieat all, and b) does not have separate collections for the different forecast hour sets (e.g. fh00-01 and fh02-48) and it includes byte range information that downstream applications can use to read small subsets of the GRIB files.There are separate collections for each product (e.g. sfc, nat, etc).
The vision here is that it would be possible to perform a query of STAC items and load the assets into xarray with a very lightweight application (no eccodes or cfgrib or Herbie) by using the GDAL GRIB driver. This is an improvement over the metadata structure in #3 which would require much more application code in order to identify the byte subsets and parse the variables.
I opted to set the hrefs as
/vsisubfile/{start_byte}_{size},/vsicurl/{grib_url}, but we could also set the hrefs to the plain old grib url and add the byte range data to the asset properties.Here is a sample of the item metadata with one asset per GRIB layer:
https://github.com/developmentseed/noaa-hrrr/blob/9920bcb730cba4a29842e570847e6105c18f922d/examples/hrrr-conus-sfc-2024-05-10T12-FH0/hrrr-conus-sfc-2024-05-10T12-FH0.json#L59-L104