MET V11.1.1 reading ECMWF ensemble grib1 files

[pratiman-91]

I want to know if it is possible to read ECMWF single grib1 file containing all ensemble members. It works for deterministic forecasts. Using GRIB_ens="MET_ENS_MEMBER_ID" does not work. At least it reads the file and finishes unsuccessfully. It is possible that I may be missing something.

I have also converted the Grib1 to Grib2 using grib_set -s edition=2, which MET doesn't like and stops with segmentation fault while reading the Grib2 file. Maybe some conflicts with the Grib library and how MET is installed?

My go-to way is using Python embeddings, which works, but it takes a lot of time to read the 51 members for each variable.

TLDR: Is it possible to read ECMWF 51 ensemble single grib1 file? If yes, how?

Thank you for your support.
-Pratiman

[CPKalb]

Hi Pratiman:

MET does include logic to differentiate between different members in a single grib file, as long as they are stored in the same way that NOAA stores their files. An example file with multiple members that MET can read can be found here. I realize this is a grib2 file, and your data is grib1. But, is the data otherwise stored similarly in your file?

This file can be read into MET's plot_data_plane, to get a plot of ensemble member 6 at the 12 hour lead time using the following command:

plot_data_plane enspost_grb2.t00z.prmsl plot.ps 'name="PRMSL"; level="L0"; GRIB_ens="+6"; lead_time="12";'

Are you reading this data into Ensemble-Stat or another MET program? If you are reading into Ensemble-Stat, you would want to use the ens_member_id variable instead of the GRIB_ens variable. Would you be able to post the configuration that is not working here so I can see what is set and which MET program you are running? You will need to give you configuration file a .txt extension to be able to post it to this discussion. Also could you post a copy of the output from running wgrib on the file? Having this information will make it easier to get the configurations correct if MET can read the file.

Thanks,
Christina

[pratiman-91]

Thank you for your fast reply.

I have attached my configuration script for METplus.

EnsembleStat_ecmwf_eps_gh_sonde.txt

Unfortunately, wgrib is not installed in our system. And would not be possible anytime soon. However, I have tried to use plot_data_plane

DEBUG 1: Start plot_data_plane by user(12889) at 2025-01-10 07:45:49Z  cmd: plot_data_plane 20241122_12.grib1 plot.ps name="GH"; level="P500"; GRIB_ens="+3"; GRIB1_ptv = 128;
DEBUG 1: Opening data file: 20241122_12.grib1
WARNING:
WARNING: MetGrib1DataFile::data_plane() -> Found 51 matches for VarInfo "GH/P500" in GRIB file "20241122_12.grib1".  Using the first match found.
WARNING:
DEBUG 1: Creating postscript file: plot.ps
DEBUG 1: Finish plot_data_plane by user(12889) at 2025-01-10 07:45:49Z

I want to select only a particular ensemble member rather than the first one.

-Pratiman

[CPKalb]

In this case there is likely either an error in the way GRIB_ens is specified, or some additional filtering is needed. It's hard to know exactly what settings are needed without having any information about the file. Would you be able to compress the file (by gzip or something similar) and attach it to this discussion so I can take a closer look. If it's too big to do this, let me know and I'll see what I can do.

Thanks,
Christina

[pratiman-91]

Please check the attachment for the grib1 file. It has only GH. The whole file was too big for GitHub. If you need the whole file then maybe I can upload somewhere else.
gh_20241122_0000_12_regular_ll.zip

Thanks
-Pratiman

[JohnHalleyGotway]

Hello @pratiman-91, I work with @CPKalb, who's out of the office this week at the AMS meeting. But I pulled down your file and took a look. Using wgrib, I see that it has 306 records in it:

wgrib gh_20241122_0000_12_regular_ll.grib1
1:0:d=24112200:GH:kpds5=156:kpds6=100:kpds7=500:TR=0:P1=12:P2=0:TimeU=1:500 mb:12hr fcst:type=Perturbed forecast 4:NAve=0
...

All are for the variable GH, for 6 different pressure levels (1000 mb, 925 mb, 850 mb, 700 mb, 500 mb, 200 mb), and 51 different members (type=Control forecast 0, type=Perturbed forecast 1 ... type=Perturbed forecast 50). So that's 6 levels * 51 members = 306 records. And you're wondering how to distinguish between the GRIB1 records for the different members.

I did a deep dive and do see what's going on.

First, let me point you to this NCEP GRIB1 documentation (https://www.nco.ncep.noaa.gov/pmb/docs/on388/appendixc.html), I note the following:

This appendix details NCEP's use of the extended GRIB PDS section (octets 41 to 86) to describe ensemble model data. Currently there are no WMO standards for including this extra information on ensemble data in GRIB1.
This is what was used when adding support for GRIB1 ensemble data.

In your file, (wgrib -PDS10 gh_20241122_0000_12_regular_ll.grib1) I see...
Octets 41, 42, 43 = 30, 1, 11 for the ensemble flag, type, and number, respectively.
And those values remain constant for all records, which means they can't be used to differentiate between records.

I notice in the wgrib -PDS10 output, that your "forecast number" (19 in the example below) is stored in the 50-th column of the PDS.

306:10435270:d=24112200:GH:kpds5=156:kpds6=100:kpds7=200:TR=0:P1=12:P2=0:TimeU=1:200 mb:12hr fcst:type=Perturbed forecast 19:NAve=0:PDS10= 0 0 106 128 98 158 255 128 156 100 0 200 24 11 22 0 0 1 12 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 30 1 11 4 11 48 48 48 49 19 51 2 0 0 0 1 52 218 226 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

This data is for the ECMWF, originating center 98 (5-th entry above), and presumably ECMWF encode ensemble metadata differently, and we haven't encountered it yet in MET. I do this line of code in the wgrib source code that's handling this data. But that logic does not exist in MET.

So what can you do at this point? In MET, the GRIB2_ipdtmpl_index (0-based) and GRIB2_ipdtmpl_val are meant as a fallback, so that we could filter by any values in the PDS. Unfortunately, those are only supported for GRIB2 data, not GRIB1.

I did try converting this from GRIB1 to GRIB2 and then used these config options to plot the result for member 25:

cnvgrib -g12 gh_20241122_0000_12_regular_ll.grib1 gh_20241122_0000_12_regular_ll.grib2
plot_data_plane gh_20241122_0000_12_regular_ll.grib2 HGT_P500_ENS_25.ps 'name="HGT"; level="P500"; GRIB2_ipdtmpl_index = 16; GRIB2_ipdtmpl_val = 25;'

So that's the rather ugly workaround...

• Convert from GRIB1 to GRIB2.
• Use MET's GRIB2_ipdtmpl_index and GRIB2_ipdtmpl_val config options to select the member of interest.

Providing direct support for ECMWF GRIB1 ensemble conventions would require additional development in MET. And to do that, we'd need to gather sample data, write up a development issue, and identify a funding source to pay for that work.

[pratiman-91]

@JohnHalleyGotway Thank you for the detailed explanation. For now, I am splitting the grib1 file by perturbationNumber and reading the resulting files separately. However, this approach duplicates the data. Ideally, it would be better to read the data directly from the source, though I understand this would be a lengthy process.

Regarding funding sources, I’m not sure if this is relevant, but the Meteorological Service Singapore (MSS) have some kind of arrangement with NCAR. I don’t know the specifics, but if it would help, I can reach out to the appropriate contact to gather more information.
-Pratiman

[JohnHalleyGotway]

Thanks @pratiman-91 for following up. I'm glad you have a workaround, but definitely agree that direct support of the GRIB1 ECMWRF GRIB1 conventions would be far preferable. I'll go ahead and mark this discussion as being answered and close it.

I'll note that I did add this topic to our (brand new) METplus Idea Catcher project board to document good development ideas prior to writing them up as issues in the correct METplus repository.