Point Stat not matching elevation levels when comparing gridded fields to point observations.

[Wesley-J-Davis]

Below are the dictionaries returned with verbosity = 10 for both fcst and ob. As you can see the level strings match, but when I try to compare said levels I get a obs_hgt

_| | | | | | | | String Value = "/p/work1/daviswj/neptune-verif/read_NRL_FLATFILE_CONVERT_CtoK.py /p/work1/daviswj/FCST/r21dv/airtmp_zht_0002.0_0000.0_glob360x181_2021060100_00960000_fcstfld"
| | | | | | | Dictionary Entry[1] ...
| | | | | | | | Name = set_attr_level
| | | | | | | | Type = StringType
| | | | | | | | String Value = "Z2"
| | | | | | | Dictionary Entry[2] ...
| | | | | | | | Name = cat_thresh
| | | | | | | | Type = ArrayType
| | | | | | | | Dict/Array Value =
| | | | | | | | | Nentries = 1
| | | | | | | | | Address = 0x19236b0
| | | | | | | | | Parent = 0x1922c50
| | | | | | | | | IsArray = true
| | | | | | | | | Dictionary Entry[0] ...
| | | | | | | | | | Name = (nul)
| | | | | | | | | | Type = ThresholdType
| | | | | | | | | | Thresh Value =
| | | | | | | | | | | type = 0
| | | | | | | | | | | value = -9999
| | | | | | | | | | | ptype = -1
| | | | | | | | | | | pvalue = -9999
| Dictionary Entry[113] ...
| | Name = obs
| | Type = DictionaryType
| | Dict/Array Value =
| | | Nentries = 1
| | | Address = 0x191c290
| | | Parent = 0xb52508
| | | IsArray = false
| | | Dictionary Entry[0] ...
| | | | Name = field
| | | | Type = ArrayType
| | | | Dict/Array Value =
| | | | | Nentries = 1
| | | | | Address = 0x151ec40
| | | | | Parent = 0x191c290
| | | | | IsArray = true
| | | | | Dictionary Entry[0] ...
| | | | | | Name = (nul)
| | | | | | Type = DictionaryType
| | | | | | Dict/Array Value =
| | | | | | | Nentries = 3
| | | | | | | Address = 0x191ad40
| | | | | | | Parent = 0x151ec40
| | | | | | | IsArray = false
| | | | | | | Dictionary Entry[0] ...
| | | | | | | | Name = name
| | | | | | | | Type = StringType
| | | | | | | | String Value = "t"
| | | | | | | Dictionary Entry[1] ...
| | | | | | | | Name = level
| | | | | | | | Type = ArrayType
| | | | | | | | Dict/Array Value =
| | | | | | | | | Nentries = 1
| | | | | | | | | Address = 0x19236f0
| | | | | | | | | Parent = 0x191ad40
| | | | | | | | | IsArray = true
| | | | | | | | | Dictionary Entry[0] ...
| | | | | | | | | | Name = (nul)
| | | | | | | | | | Type = StringType
| | | | | | | | | | String Value = "Z2"
_

The code runs fine without breaking and produces empty output files save for headers. The debug information for the matched pairs goes as follows:

DEBUG 2: Processing airtmp/Z2 versus t/Z2, for observation type RAOB, over region TROPICS, for interpolation method NEAREST(1), using 0 matched pairs.
DEBUG 2: Number of matched pairs = 0
DEBUG 2: Observations processed = 13219
DEBUG 2: Rejected: station id = 0
DEBUG 2: Rejected: obs var name = 0
DEBUG 2: Rejected: valid time = 0
DEBUG 2: Rejected: bad obs value = 0
DEBUG 2: Rejected: off the grid = 0
DEBUG 2: Rejected: topography = 0
DEBUG 2: Rejected: level mismatch = 13219
DEBUG 2: Rejected: quality marker = 0
DEBUG 2: Rejected: message type = 0
DEBUG 2: Rejected: masking region = 0
DEBUG 2: Rejected: bad fcst value = 0
DEBUG 2: Rejected: bad climo mean = 0
DEBUG 2: Rejected: bad climo stdev = 0
DEBUG 2: Rejected: mpr filter = 0
DEBUG 2: Rejected: duplicates = 0
DEBUG 2:

The config file information is set up as follows:

fcst = {
field = [

    {
    name       = "/p/work1/daviswj/neptune-verif/read_NRL_FLATFILE_CONVERT_CtoK.py /p/work1/daviswj/FCST/r21dv/airtmp_zht_0002.0_0000.0_glob360x181_2021060100_00960000_fcstfld";
      set_attr_level     = "Z2";
      cat_thresh = [ NA ];
       }


  ];

}

obs = {
field = [

       {
      name       = "t";
      level      = ["Z2"];
      cat_thresh = [ NA ];
        }


  ];

}

I have placed all relevant files in DAVIS_DATA.

The technique used to convert the h5 files for the obs in to MET readable format came about from work done on the following discussion:

#2391

I took that reader and adapted it to a differently formatted observation file and am using the same point obs calling format, e.g.:

PYTHON_NUMPY="/path/to/test_h5_embed.py /path/to/file variable"

I just need help matching the obs_hgt portion of the ob to the level of the gridded file. Thank you.

[DanielAdriaansen]

Hi @Wesley-J-Davis, thanks for writing in and providing all of your files. I was able to reproduce your error locally. Here is the command I used for testing:

/usr/local/met-11.0.1/bin/point_stat PYTHON_NUMPY 'PYTHON_NUMPY=/home/dadriaan/tmpdisc/2426/DAVIS_DATA/test_h5_embed.py /home/dadriaan/tmpdisc/2426/DAVIS_DATA/NAF_sfc_2021060500.h5 temperature' PointStatConfig_IODA_r21dv_2021060500_000096 -outdir ./test_output -v 6

I see that in the obs dictionary, you are telling PointStat that you want to use observations at level Z2. This implies a "height of 2m":

      level      = ["Z2"];

For this to work, you have to make sure the height of the observations you expect to be at 2m is set to 2m, but in your Python script test_h5_embed.py, I see that you are setting the hgt column to 0m for all observations on line 227:

               'hgt' : [0]* len(elv),

After changing that line to:

               'hgt' : [2]* len(elv),

I got farther along, but now received this error:

DEBUG 2: Observations processed    = 13219
DEBUG 2: Rejected: station id      = 0
DEBUG 2: Rejected: obs var name    = 0
DEBUG 2: Rejected: valid time      = 0
DEBUG 2: Rejected: bad obs value   = 0
DEBUG 2: Rejected: off the grid    = 0
DEBUG 2: Rejected: topography      = 0
DEBUG 2: Rejected: level mismatch  = 0
DEBUG 2: Rejected: quality marker  = 0
DEBUG 2: Rejected: message type    = 13219

Now there are no level mis-matches but all the observations don't match the type. In your Python script on line 51, you are setting the "type" of observation being used:

typ = list(data.keys())[0]

For the command you sent, the value of typ in your script is SYNOP. But in your PointStat config file on line 103, you are specifying the message_type as RAOB:

message_type   = ["RAOB"];

If I switch this to SYNOP:

message_type   = ["SYNOP"];

then I am ultimately successful at matching all observations:

DEBUG 3: Number of matched pairs   = 13219
DEBUG 3: Observations processed    = 13219

PLEASE NOTE: These changes assume ALL of your observations are at 2m (I am forcing the hgt column to a value of 2 for all observations). If some of your observations are at other levels like 10m, you must modify your Python code to accurately assign the hgt of the SYNOP observation otherwise PointStat will use all data.

Please let us know if this answers your questions or you have any follow up questions!

[Wesley-J-Davis]

Thank you!