diff --git a/DTC/metviewer_plotting/README b/DTC/metviewer_plotting/README deleted file mode 100644 index 06010aef2c..0000000000 --- a/DTC/metviewer_plotting/README +++ /dev/null @@ -1 +0,0 @@ -Instructions currently located at: https://docs.google.com/document/d/1fkmAer417kdrYrLF28-SeRwf2IzPE4ScSleO7oe6eZ4/edit?usp=sharing diff --git a/DTC/metviewer_plotting/README.txt b/DTC/metviewer_plotting/README.txt new file mode 100644 index 0000000000..6c3ed70c86 --- /dev/null +++ b/DTC/metviewer_plotting/README.txt @@ -0,0 +1,484 @@ +Instructions currently located at: https://docs.google.com/document/d/1fkmAer417kdrYrLF28-SeRwf2IzPE4ScSleO7oe6eZ4/edit?usp=sharing + +IMPORTANT: +Currently, the automated METviewer plotting capability described here only works on NCAR’s mohawk machine, so you’ll first have to log into mohawk. + + +Introduction +This document describes the automated capability in the ???[a] branch DTC’s fork of the UFS SRW App repository to generate verification plots from the command line using the METviewer batch plotting script (aka METviewer batch mode). Here, by “automated”, we mean (1) without the need for the user to first manually create the xml files that METviewer reads in, and (2) with the capability to generate many vx plots via a single command, where “many” refers to multiple combinations of vx metric, forecast field, forecast level, and (if required by the metric) threshold, with one vx plot generated per such combination. + + +Note that this capability is meant to be used for plotting verification output from the SRW App that has already been manually loaded into a METviewer database on a host on which METviewer is installed. METviewer is not installed on most NOAA high-performance computing (HPC) machines (hosts), apparently because it can introduce security issues. Thus, the step of creating a database from SRW verification output (which consists of the .stat files generated by METplus) cannot be automated as part of the automated METviewer plotting capability and is not described here. + + +In this document, we let $BASEdir (in bash notation) refer to the directory in which the SRW App will be cloned. This must be a directory to which the user has read and write permissions, e.g. + + + BASEdir = /path/that/user/can/access + + +Also, we let $HOMEdir refer to the top-level directory in which the SRW App is cloned. By default, this will be a subdirectory named ufs-srweather-app under $BASEdir, i.e. + + + HOMEdir = $BASEdir/ufs-srweather-app + + +Finally, we let $DTCdir refer to the subdirectory DTC under $HOMEdir that contains various DTC-specific codes and other files/directories that will not be committed to the authoritative SRW repository. Thus, + + + DTCdir = $HOMEdir/DTC + + +Running the Automated METviewer Plotting Capability +This section describes the procedure for setting up and running the automated METviewer plotting capability on NCAR’s mohawk machine. + + +The initial set of steps to set up the proper environment are as follows: + + +1. Log into mohawk. + + +2. Change location to $BASEdir: + + +> cd $BASEdir + + +3. Clone the branch feature/generate_metviewer_xmls_ams from Gerard’s fork of the SRW App: + + +> git clone -b feature/generate_metviewer_xmls_ams https://github.com/gsketefian/ufs-srweather-app.git + + +(This branch will be merged into the ???[b] branch in the DTC’s fork of the SRW App repository in the near future.) Once this is complete, there will be a subdirectory under $BASEdir named ufs-srweather-app containing the SRW App, and there will be a subdirectory under ufs-srweather-app named DTC as described above. The full paths to these subdirectories are given by $HOMEdir and $DTCdir, respectively. + + +4. Check out all the external repositories needed by the SRW App (although only the workflow-tools repo is needed here): + + +> cd $HOMEdir +> ./manage_externals/checkout_externals + + +This step may take a few minutes to complete. + + +5. Source the script that sets up conda, e.g. from Gerard’s home directory (you can also install this yourself, or maybe use one installed on mohawk, but that has not been tested): + + +> source ~ketefian/conda/etc/profile.d/conda.sh + + +This step is necessary for activating the workflow_tools environment in the next step. + + +6. Activate the workflow_tools environment: + + +> conda activate workflow_tools + + +7. Change directory to the location of the python scripts and other files that comprise the automated METviewer plotting capability: + + +> cd $DTCdir/metviewer_plotting + + +Generating a Single Verification Plot +To generate a single verification plot, use the python script make_single_mv_vx_plot.py located in the directory $DTCdir/metviewer_plotting. Help for this script can be accessed as follows: + + +> cd $DTCdir/metviewer_plotting +> ./make_single_mv_vx_plot.py –-help + + +For example, to call the METviewer batch plotting script on mohawk to create a rank histogram (rhist) plot of composite reflectivity (refc) for >= 30 dBZ for the models with short names href, gdas, and gefs in the database mv_gefs_gdas_rtps_href_spring_2022, use: + + +> ./make_single_mv_vx_plot.py \ + --mv_host mohawk \ + --mv_database mv_gefs_gdas_rtps_href_spring_2022 \ + --model_names_short href gdas gefs \ + --model_colors red blue green \ + --fcst_init_info 2022050100 6 24 \ + --fcst_len_hrs 36 \ + --vx_metric rhist \ + --fcst_field refc \ + --fcst_level L0 \ + --threshold ge30dBZ + + +This command will create (among others) the following four output files in the default location $HOMEdir/../expts_dir/mv_output/plots: + + +* plot_rhist_REFC_href_gdas_gefs.xml +* plot_rhist_REFC_href_gdas_gefs.html +* plot_rhist_REFC_href_gdas_gefs.png +* plot_rhist_REFC_href_gdas_gefs.log + + +The xml is generated first. This is then passed to the METviewer batch plotting script, which generates the html (an intermediate file which we are not concerned with here) and the png image file (which contains the vx plot). The screen and error (stdout and stderr) output from the METviewer batch script is saved in the log file. + + +Note the following: + + +1. The argument –-mv_host specifies the host on which METviewer is installed and on which the python scripts of the automated METviewer plotting capability described here are running. The host name passed to this argument must have an entry in the yaml host configuration file. This configuration file is by default mv_hosts.yaml but can be changed using the -–mv_host_config_fp option. + + +2. The argument –-mv_database specifies the METviewer database that exists on the current host and which contains the vx data to be plotted. The database name passed to this argument must have an entry in the yaml database configuration file. This is by default mv_databases.yaml but can be changed using the -–mv_database_config_fp option. + + +3. The argument –-model_short_names specifies the short names of the forecast models in the database to include in the plot. These short names are for convenience because the actual (long) names for the models in the METviewer database can be quite long. Thus, these short names are introduced for use on the command line as well as in the plot legends. For each database, the mapping between the short model names and the (long) names used in the database is (should be) given in the database configuration file. All short names passed to this argument must appear in that configuration file under the entry for the database specified on the command line. + + +4. The option --model_colors specifies the color to use for each forecast model in the plot. If this is not specified, the available colors in the yaml file valid_vx_plot_params.yaml that specifies valid values for various vx plotting parameters. In this case, the colors are assigned to the models in the order they appear in this file. If more models to plot are specified than there are colors specified/available, the script stops with an appropriate error message. + + +5. The user can specify an alternate location for the output files using the --output_dir option. For example, specifying + + +--output_dir /some/non/default/location + + +will result in the xml, html, png, and log files being placed in the directory /some/non/default/location/plots. Note that the files are placed in a subdirectory named plots under the directory passed to the --output_dir option. + + +6. The argument --fcst_init_info provides information on the initialization times of the forecasts. It takes 3 integers, defined as follows: + + +--fcst_init_info YYYYMMDDHH num_fcsts fcst_init_intvl_hrs + + +Here, YYYYMMDDHH is the starting time of the first forecast, num_fcsts is the total number of forecasts, and fcst_init_intvl_hrs is the number of hours between the initialization times of the forecasts. For example, + + +--fcst_init_info 2022050100 6 24 + + +represents a set of 6 forecast initialization times that are separated by 24 hours, with the first one at 00Z on May 1, 2022. This corresponds to the following 6 forecast initialization times: + + +2022050100 2022050200 2022050300 2022050400 2022050500 2022050600 + + +7. The argument --fcst_len_hrs specifies the forecast length, in hours. All forecasts (i.e. for all initialization times and ensemble members) are assumed to have the same length. + + +8. The argument --vx_metric specifies the verification (vx) metric for which to generate a plot. Valid vx metric names that may be passed to this argument are given in the yaml valid-values configuration file valid_vx_plot_params.yaml. + + +9. The argument --fcst_field specifies the field for which to generate a plot. Valid forecast field names that may be passed to this argument are given in the yaml valid-values configuration file valid_vx_plot_params.yaml. + + +10. The argument --fcst_level specifies the level at which (or the accumulation interval for which) to generate a plot of the specified metric for the specified forecast field. Valid level or accumulation values (corresponding to each valid forecast field) that may be passed to this argument are given in the yaml valid-values configuration file valid_vx_plot_params.yaml. + + +11. The argument --threshold specifies the threshold for which to generate a plot. Valid threshold values for the metric, field, and level specified on the command line are given in the yaml valid-values configuration file valid_vx_plot_params.yaml. Note that some vx metrics require a threshold while others don’t. Whether or not a metric requires a threshold is specified in valid_vx_plot_params.yaml. If the --threshold argument is specified for a metric that does not need it, an informational message is printed and the threshold is ignored. + + +12. When plotting one of the verification metrics bias (bias or mean error) and fbias (frequency bias), the default behavior is to include in the plots the curves for the ensemble means (in addition to the curves for the individual members). To leave out the ensemble mean curves, specify the --no-incl_ens_means flag. + + +Generating Multiple Verification Plots[c][d] +To generate multiple verification plots, use the python script make_multi_mv_vx_plots.py (also located in the directory $DTCdir/metviewer_plotting). Help for this script can be accessed as follows: + + +> cd $DTCdir/metviewer_plotting +> ./make_multi_mv_vx_plots.py –-help + + +This script in turn calls make_single_mv_vx_plot.py to generate a plot for each specified combination of verification metric, forecast field, forecast level or accumulation, and (if required by the metric) threshold. + + +Most of the information needed by make_multi_mv_vx_plots.py is in a yaml plot configuration file that can be specified via the --plot_config_fp option. The default name of this configuration file is multi_plot_config.default.yaml. This file provides the following information to make_multi_mv_vx_plots.py (a more detailed description of the contents of this configuration file are given later below): + + +* The host on which METviewer is installed and the METviewer batch script will be running. +* The name of the METviewer database from which to obtain the vx data to plot. +* The short names of the forecast models in that database to plot (the mapping between these short names and the (long) model names in the database is specified in mv_databases.yaml). +* The forecast initialization information (consisting of the date and time of the first initialization, the number of initializations, and the time interval in hours between initializations). +* The forecast length, in hours; all forecasts (i.e. for all initialization times and ensemble members) are assumed to have the same length. +* All combinations of verification metric, forecast field, forecast level or accumulation, and (if applicable) threshold for which to generate vx plots. + + +To generate a set of vx plots using the above information as specified in the default plot configuration file multi_plot_config.default.yaml, issue the command: + + +> ./make_multi_mv_vx_plots.py + + +This will generate a set of xml, html, png, and log files (one such set per metric-field-level-threshold combination specified in the plot configuration file) in the default location $HOMEdir/../expts_dir/mv_output/plots. + + +Note the following: + + +1. The location where the output files are placed can be changed using the --output_dir option. For example, + + +> ./make_multi_mv_vx_plots.py --output_dir /some/non/default/location + + +will result in the png, xml, html, and log files being placed in the directory /some/non/default/location/plots. Note that the output files are not placed directly under the directory passed to this option but in a subdirectory under it named plots. + + +2. The option –-preexisting_dir_method specifies the method to use if a preexisting output directory is encountered, i.e. if the directory in which the script will place its output already exists. The argument passed to this option can be one of rename, delete, and quit. The default is rename, which will keep a copy of the preexisting directory by renaming it (by adding a suffix to the directory name based on the current date and time). Specifying delete will cause the preexisting directory to be deleted and a new one created, and specifying quit will cause the script to quit if it encounters a preexisitng output directory. + + +3. To use a plot configuration file other than the default, use the option --plot_config_fp. For example, + + +> ./make_multi_mv_vx_plots.py --plot_config_fp multi_plot_config.alternate.yaml + + +will look for the plot configuration file multi_plot_config.alternate.yaml instead of the default one. The contents of this non-default configuration file must be in the same format as those in the default file multi_plot_config.default.yaml, but the database name, model short names, forecast initialization information, forecast length, and the set of metric-field-level-threshold combinations for which to generate vx plots can of course be different. + + +4. Verification metrics for which plots are generated can be limited to a subset of the ones specified in the plot configuration file using the --incl_only_metrics option. For example, specifying + + +> ./make_multi_mv_vx_plots.py --incl_only_metrics bias ss + + +will generate plots of the metrics bias (bias aka mean error) and ss (spread-skill score) but not for any other metrics in the plot configuration file. If a metric passed to this option is not included in the plot configuration file, a warning message is printed and no plots are generated for that metric. + + +5. Verification metrics specified in the plot configuration file can be excluded from plot generation using the --excl_metrics option. For example, specifying + + +> ./make_multi_mv_vx_plots.py --excl_metrics auc brier + + +will prevent the script from generating plots or the metrics auc (area-under-the-curve) and brier (Brier score), but plots will be generated for all other metrics in the plot configuration file. If a metric passed to this option is not included in the plot configuration file, a warning message is printed. Note that this option cannot be used together with --incl_only_metrics; for simplicity, only one of these two options can appear on the command line. + + +6. Forecast fields, forecast levels, and thresholds for which vx plots will be generated can be limited to subsets of the ones specified in the plot configuration file using options analogous to the one for metrics. These options are: + + +--incl_only_fields ... +--incl_only_levels ... +--incl_only_threshes ... + + +For example, to include in the plotting only the fields apcp and refc at only levels 03h and L0 and only for thresholds ge6.35mm and ge20dBZ, use + + +> ./make_multi_mv_vx_plots.py \ + --incl_only_fields apcp refc \ + --incl_only_levels 03h L0 \ + --incl_only_threshes ge6.35mm ge20dBZ + + +If a field, level, or threshold passed to these options is not included in the plot configuration file, a warning message is printed and no plots are generated for that field, level, or threshold. + + +7. Forecast fields, forecast levels, and thresholds specified in the plot configuration file can be excluded from plot generation using options analogous to the one for metrics. These options are: + + +--excl_fields ... +--excl_levels ... +--excl_threshes ... + + +For example, to exclude from plotting the fields apcp and refc, the levels 03h and L0, and the thresholds ge6.35mm and ge20dBZ, use + + +> ./make_multi_mv_vx_plots.py \ + --excl_fields apcp refc \ + --excl_levels 03h L0 \ + --excl_threshes ge6.35mm ge20dBZ + + +If a field, level, or threshold passed to these options is not included in the plot configuration file, a warning message is printed. Note that, as with the --incl_only_metrics and --excl_metrics options, for simplicity we do not allow the --incl_only_fields, --incl_only_levels, and --incl_only_threshes options to be used together with the +--excl_fields, --excl_levels, and --excl_threshes options, respectively. + + +8. The --incl_only_... and --excl_... options can be used together in the same command as long as they are not specified for the same plotting parameter (where by plotting parameter, we mean metric, field, level, or threshold). For example, + + +> ./make_multi_mv_vx_plots.py \ + --excl_metrics bias ss \ + --incl_only_fields apcp refc \ + --excl_levels 03h L0 \ + --incl_only_threshes ge6.35mm ge20dBZ + + +9. Since the screen output that make_multi_mv_vx_plots.py generates can be extensive, it can be redirected to a log file using the --log_fp option. For example, to redirect output from the make_multi_mv_vx_plots.py script to the file mv_batch_plotting.log (in the same directory as the script) while the script runs in the background, use + + +> ./make_multi_mv_vx_plots.py --log_fp mv_batch_plotting.log & + + +Note the ampersand at the end of the command to send it to the background. + + +10. The flag –-make_metric_subdirs causes the output files for each vx metric to be placed in a separate metric-specific subdirectory under the output directory (instead of the output files for all metrics being placed in the plots subdirectory under the output directory). These metric-specific subdirectories are named after the metrics they correspond to. For example, if this flag is specified on the command line and if the metrics to be plotted are reliability and rank histogram (e.g. as specified in the plot configuration file via the entries rely and rhist), and if the output directory (set either by default or on the command line via the --output_dir option) is ${output_dir}, then the png image files (as well as the xml, html, and log files) for all reliability plots (where by “all reliability plots” we mean reliability plots for all field, level, and threshold combinations specified in the plot configuration file) are placed in ${output_dir}/rely, and those for rank histogram (which doesn’t require a threshold) are placed in ${output_dir}/rely. Without the –-make_metric_subdirs option on the command line, the output files for reliability and rank histogram would all be placed in ${output_dir}/plots. + + +11. The flag –create_ordered_plots causes a subdirectory named ordered_plots to be created under the output directory and copies of all the plots (the png files) to be placed there but renamed such that when listed alphabetically, they appear in the order specified in the plot configuration file. The renaming is done by adding the prefix p###_ to the file name, where ### is a 3-digit integer that is set to 001 for the first metric-field-level-threshold combination appearing in the plot configuration file, to 002 for the second such combination in the plot configuration file, etc. For example, if the combination of metric rely, field apcp, level (really accumulation in this case) 06h, and threshold ge12.7mm is the 7th one appearing in the plot configuration file (where we count over thresholds first, then levels then fields, and finally metrics), then the usual image file generated for this combination would be at + + +${output_dir}/plots/plot_rely_APCP06h_ge12.7mm_href_gdas_gefs.png + + +With the –create_ordered_plots flag specified on the command line, there would be a copy of this image file at + + +${output_dir}/p007_plot_rely_APCP06h_ge12.7mm_href_gdas_gefs.png + + +The –create_ordered_plots flag is useful if the user intends to place all the image files generated in a single pdf file in the order that the corresponding metric-field-level-threshold combinations are listed in the plot configuration file (e.g. for the purposes of preparing a report or a presentation). The convenience comes from the fact that all the png files are in a single directory (${output_dir}/ordered_plots) and alphabetically ordered in the way the user would like them to appear in the pdf file. Adobe Acrobat’s Create PDF functionality can then be used to import all the images in the desired order and create a single pdf file. + + +For example, to generate frequency bias (fbias) and reliability (rely) plots, place the output files in /some/new/location/plots, and redirect the screen output to mv_plot_generation.log (which will be created in the same directory as the make_multi_mv_vx_plots.py script because we are not using a full path in this case), use: + + +> ./make_multi_mv_vx_plots.py \ + --incl_only_metrics fbias rely \ + --output_dir /some/new/location \ + --log_fp mv_plot_generation.log + + +Code Description +Overview +The files in the DTC’s fork of the SRW App needed for the automated METviewer verification plotting capability are located in $DTCdir/metviewer_plotting and consist of the following: + + +* Two python scripts named make_single_mv_vx_plot.py and make_multi_mv_vx_plots.py. + + +* A set of four yaml configuration files that either one or both of the python scripts read in. These are named: + * mv_hosts.yaml – the METviewer hosts configuration file + * mv_databases.yaml – the METviewer databases configuration file + * valid_vx_plot_params.yaml – the valid-values configuration file + * multi_plot_config.default.yaml – the default plot configuration file for the script make_multi_mv_vx_plots.py + + +* A set of jinja2 templates for generating METviewer xmls in the parm subdirectory under $DTCdir/metviewer_plotting. + + +The Python Scripts +The script make_single_mv_vx_plot.py is used to generate a single verification plot for a given combination of a verification metric, forecast field, a level or accumulation for that forecast field, and, if necessary, a threshold for the forecast field (some vx metrics require a threshold while others don’t). The main steps it executes are: + + +1. Read in the command line arguments and accordingly set the values of the jinja2 variables that appear in one of the jinja2 xml templates described above. + + +2. Generate a METviewer xml by calling the python function set_template (part of the workflow-tools repository that the SRW App clones), passing to it the values of the jinja2 variables evaluated in the previous step along with the path to one of the jinja2 xml templates. + + +3. Generate a verification plot (as a png image file) by calling the METviewer batch plot generation script and passing to it the xml generated in the previous step. + + +Help for make_single_mv_vx_plot.py can be accessed as follows: + + +> cd $DTCdir/metviewer_plotting +> ./make_single_mv_vx_plot.py –-help + + +Besides the command line arguments described in this help, this script reads in the yaml configuration files mv_hosts.yaml, mv_databases.yaml, and valid_vx_plot_params.yaml. + + +The script make_multi_mv_vx_plots.py automates the creation of multiple verification plots. It does this by looping through a specified set of combinations of verification metric, forecast field, forecast level or accumulation, and (if applicable) threshold, and, for each such combination, calling make_single_mv_vx_plot.py to generate a plot. Help for make_multi_mv_vx_plots.py can be accessed as follows: + + +> cd $DTCdir/metviewer_plotting +> ./make_multi_mv_vx_plots.py –-help + + +Besides the command line arguments described in this help, this script reads in the yaml configuration files mv_databases.yaml, valid_vx_plot_params.yaml., and either the default plot configuration file multi_plot_config.default.yaml or a custom one created by the user (and specified on the command line via the --plot_config_fp option). + + +The YAML Configuration Files +The yaml file mv_hosts.yaml defines host-related METviewer parameters. The top-level entries consist of a set of host names that may be specified on the command line. Under each name are listed the following set of parameters (for that host): + + +* user +Name of the METviewer user. +* password +Name of the METviewer user’s password. +* Rscript_fp +Path to R-script executable. +* R_tmpl_dir +Path to directory containing R templates. +* R_work_dir +Path to R work directory. +* mv_batch_fp +Path to METviewer batch plotting script. + + +If the host name passed to the python plotting scripts (either on the command line or via yaml files) is not in the set of top-level entries in this yaml file, then the scripts will be unable to generate plots. + + +The yaml file mv_databases.yaml defines database-related parameters. It consists of a set of database names (the top-level entries in the file), with each name followed by a (1) a set of valid thresholds for the database and (2) a set of forecast model dictionaries – one per model that the database contains. Each such forecast model dictionary in turn contains the following keys: + + +* long_name +The long name of the forecast model, i.e. the name as it appears in the METviewer database. +* short_name +A short name for the model that can be used to refer to it on the command line and in the vx plots (e.g. in the plot legends). This is for convenience because the model names in the METviewer database can be quite long. +* num_ens_mems +The number of ensemble members that the forecast model contains in the METviewer database. + + +If the database name passed the python plotting scripts (either on the command line or via yaml files) is not in the set of top-level entries in this yaml file, then the scripts will be unable to generate plots. (Note that the databases included in this yaml file do not all have to be on the same host; they may be located on various hosts on which METviewer is installed.) + + +The yaml file valid_vx_plot_params.yaml defines valid values for various vx plotting parameters that generally do not need to be changed by the user. The top-level entries in this file are: + + +* valid_vx_metrics +The set of valid verification metrics (e.g. auc, bias, rely) that can be passed on the command line to the python plotting scripts along with the long names of the metrics and whether or not they require a threshold. +* valid_fcst_fields +The set of valid forecast field names that can be passed on the command line to the python plotting scripts (e.g. apcp, dpt, refc, tmp, wind), and, for each one, its long name, list of valid units, and list of valid levels or accumulations (i.e. accumulation time intervals). + * valid_fcst_levels_to_levels_in_db +The mapping between the valid forecast levels/accumulations specified for the fields and the corresponding string that METviewer uses to identify that level/accumulation in the xmls. Thus, whenever a new level is added to a field under the valid_fcst_fields entry above, it must also be added to this mapping. + * avail_mv_colors_codes +The set of available colors that can be specified on the command line (one per model that is to be included in the plot) along with each color’s hex code and the hex code of the light version of that color (e.g. a hex code for blue and for light blue). The maximum number of models that can be included in the vx plots is equal to the number of colors specified here; if more models need to be included in the plots, then new colors must be added to this list. + + +The yaml file multi_plot_config.default.yaml is the default plot configuration file read in by the script make_multi_mv_vx_plots.py. This configuration file contains the following information: + + +* mv_host +Name of the host on which METviewer will be running. This must appear in the METviewer host configuration file mv_hosts.yaml. +* mv_database +Name of the METviewer database containing the verification data from which to generate plots. This must appear in the METviewer database configuration file mv_databases.yaml. +* model_names_short +Short names of the models to include in each generated plot. These short names must exist in the METviewer database configuration file mv_databases.yaml under the database specified by mv_database. +* fcst_init_info +Information on forecast initialization times. This consists of the following parameters: +* fcst_init_time_first +The initialization time of the first forecast, in the format YYYYMMDDHH. +* num_fcsts +The number of forecasts. +* fcst_init_intvl_hrs +The interval, in hours, between forecast initialization times. +* fcst_len_hrs +Length of the forecasts, in hours. All forecasts are assumed to have the same length. +* metrics_fields_levels_threshes +A set of verification metrics, with each metric containing a set of forecast fields to plot, with each field containing a set of forecast levels or accumulations, and with each level or accumulation containing a list of thresholds (which should be empty if the metric does not require a threshold). These metrics, forecast fields, and forecast levels/accumulations must all appear in the valid-values configuration file valid_vx_plot_params.yaml. + + +The Jinja2 Templates +Currently, there are four jinja2 templates in the directory $DTCdir/metviewer_plotting/parm for generating METviewer xmls. These are: + + +* auc_brier.xml – for generating AUC (Area Under the Curve) and Brier score xmls +* bias_fbias.xml – for generating bias and frequency bias xmls +* rely_rhist.xml – for generating reliability and rank histogram xmls +* ss.xml – for generating spread-skill xmls + + +The script make_single_mv_vx_plot.py uses information passed to it on the command line or via yaml configuration files to evaluate the jinja2 variables appearing in one of these templates. It then passes these values to the appropriate template to generate an xml that is in turn passed to the METviewer batch plotting script to generate a vx plot. + + +[a]Branch name TBD. +[b]Branch name TBD. +[c]As a note, I had to change two things in multi_plot_config.default.yaml: +1) mv_database_name --> mv_database +2) model_names --> model_names_short +[d]Thanks for catching that. I had forgotten to push this change to the repo. I just did, and it should work now.