Ensemble stat failure - NumArray::operator[](int) const

[Wesley-J-Davis]

ERROR :
ERROR : NumArray::operator const -> range check error ... Length = 0, i = 0
ERROR :

This failure occurred while comparing a 20 member ensemble of air temp at 2m using self-analysis (a control member of the ensemble) in the ensemble stat tool.

the file ens-stat-error.tar contains the data and scripts required to reproduce this error and can be found on your local ftp at DAVIS_DATA/ens-stat-error.tar

I am working with MET 12.0.0-beta1.

This code was written to run gen-ens-prod on a suite of ensemble files, then run ensemble stat on those same ensemble files, and then verify the gen-ens-prod output against a control member of the ensemble using grid stat.

I understand that with version 11's release, ensemble stat works differently now and one can skip the grid-stat steps and make everything directly in ensemble stat now.

The issue I am running into while trying to do just that is how to use the prob_cat_thresh and prob_pct_thresh variables in this context.

I am also unsure if the ens_ssvar_bin_size and ens_phist_bin_size configurations are correct, maybe they are causing the issue?

The template config file that is manipulated for this purpose can be found in the tar file at /NAVGEM_ensemble/config/Standard_EnsembleStatConfig.

[JohnHalleyGotway]

@Wesley-J-Davis my apologies for the long delay in following up on this discussion. Thanks for posting sample data to demonstrate. I am trying to work through what you sent to replicate and isolate the problem.

While you reference a runtime error from Ensemble-Stat, I do not actually see one in the log_ens_stat.txt log file. But I do notice that that file ends abruptly after these lines:

DEBUG 2: Subsetting pairs into 10 climatology bins.
DEBUG 3: Using 7078 of 65160 pairs for climatology bin number 1.

So my guess is that this file contains standard out, but not standard error... the stream to which error messages are written. One suggestion is that you could start using the -log command line option to write log files with both standard out and standard error. But I'll assume that I'm looking for an error message after those lines and will get to work trying to replicate it.

Thanks!

[JohnHalleyGotway]

@Wesley-J-Davis, since you didn't send me ensemble member output files, perhaps it's not the Ensemble-Stat tool I should be debugging? Maybe it's the call to Grid-Stat described in this log_grid_stat_freq.txt log file which also end abruptly with these lines:

DEBUG 3: Parsed is_u_wind = "false"
DEBUG 3: Parsed is_v_wind = "false"

I tested the following call to Grid-Stat using both version 11.1.0 and the latest 12.0.0 development version:

/nrit/ral/met-11.1.0/bin/grid_stat \
NAVGEM-ensemble/OUT_ENS_summer2014/airtmp_zht_0002_000_2014071100/ensemble_stat_20140711_000000V_ens.nc \
PYTHON_NUMPY \
NAVGEM-ensemble/OUT_ENS_summer2014/airtmp_zht_0002_000_2014071100/Standard_GridStatConfig_freq_NAVGEM_2014071100_summer2014_v11.1.0 \
-outdir /p/work2/daviswj/NAVGEM-ensemble/OUT_ENS_summer2014/airtmp_zht_0002_000_2014071100 \
-v 3 

Both runs produce the same error message:

ERROR  : 
ERROR  : check_prob_thresh() -> When verifying a probability field, you must select at least 3 thresholds beginning with 0.0 and ending with 1.0 (current setting: >=0).
ERROR  : Consider using the ==p shorthand notation for bins of equal width.
ERROR  : 

After updating the fcst dictionary like this:

< 	 cat_thresh = [  >=0  ];
---
> 	 cat_thresh = [ ==0.1  ];
92c92

And making lots of changes to paths and commenting out the polyline mask files I don't have, I did get a successful run.

So I was not able to replicate the NumArray::operator const error message you described using the data you provided.

Can you please clarify whether its coming from Ensemble-Stat or Grid-Stat? If it's Ensemble-Stat, I'll need the actual ensemble member data in order to replicate it.

Thanks! And sorry again for the delay in jumping into these details!

[Wesley-J-Davis]

Don't worry about the delay there's lots of tickets out there I wasn't expecting a 24hr turnaround.

I was trying to set the prob_cat_thresh and prob_pct_thresh in the ensemble stat portion and came across the NumArray::operator const error.

Searching that I came across this:

dtcenter/MET#2208

More searching on the matter led me to this:

dtcenter/MET#1259

Which seemed to indicate that I'd be able to produce the pstd files straight from ensemble stat without needing the grid stat steps to verify the ensemble. I was just trying to save on computing resources.

So I went ahead and did this and yes I was able to produce pstd files by setting the pstd flag = TRUE in the output file type section of the config. I had to abandon using the prob_cat_thresh and prob_pct_thresh dictionary entries though because they kept throwing that error.

So I made those pstd files from ensemble stat and when I compared it to the pstd file created in the grid stat ensemble frequency run, the results were wildly off.

Reading through that second link further I think I may have not set the ensemble_flag { freq=TRUE } and so it was not doing the comparison I expected.

I'll do some further testing on my side and if I hit a snag I'll upload what I have and try to make it less difficult for you to edit to work on your end.

[Wesley-J-Davis]

So my issue with ensemble stat is so:

When I set up my fcst dictionary to use the prob_cat_thresh dictionary entry, it requires me to set up the obs dictionary too with that parameter too. If I don't I get this error:

ERROR :
ERROR : EnsembleStatVxOpt::process_config() -> The number of forecast (5) and observation (0) probability category thresholds in "prob_cat_thresh" must match.
ERROR :

When I set up prob_cat_thresh in the obs dictionary to match the fcst dictionary though, I get this error:

ERROR :
ERROR : NumArray::operator const -> range check error ... Length = 0, i = 0
ERROR :

I included configs for both the ensemble stat run (which failed) and grid stat ensemble frequency run (which worked fine).

Standard_EnsembleStatConfig_2023070600_ens-stat-test.txt
Standard_GridStatConfig_freq_NAVGEM_2023070600_ens-stat-test.txt

According to dtcenter/MET#1259 if you leave the prob_cat_thresh part blank or left out, it uses the climo bins, which does work. But using my own thresholds is what is throwing the error.

[JohnHalleyGotway]

@Wesley-J-Davis I did some more digging, ran as similar a case as I could, and was able to replicate the runtime error you're seeing:

DEBUG 2: Subsetting pairs into 10 climatology bins.
ERROR  : 
ERROR  : NumArray::operator[](int) const -> range check error ... Length = 0, i = 0
ERROR  : 

I get this error when I specify data in the climo_mean and climo_stdev dictionaries, but it goes away when I reset file_name = [ ]; for both.

So that's one quick fix to get around this issue. Just remove the climatology data from the configuration, and you should see probabilistic output successfully written.

Unfortunately, the underlying logic for how the climatological mean and standard deviation data is used is rather complex.

In this specific case, Ensemble-Stat is computing an Nx2 probabilistic contingency table for the ensemble-derived probabilities. And it's also computing an Nx2 probabilistic contingency table for the CLIMATOLOGY-DERIVED probabilities. The climo contingency table is used when computing some of the PSTD statistics. Anyway, that's where the runtime error occurs... when processing the climo-derived probabilities. I do actually see a one-line fix to make this runtime error go away.

But I'd like to do a bit more digging and testing before summarizing it into a GitHub bugfix issue.