[Bug]: Investigate differences in plots produced with zppy test using Xarray/xCDAT codebase
#906
Labels
bug
Bug fix (will increment patch version)
Milestone
Comments
|
@forsyth2 and @tomvothecoder thank you for reporting this issue. Some results are definitely off. example https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/test-346-20241210/v3.LR.historical_0051/image_check_failures_comprehensive_v3/e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_1987-1988/lat_lon/ERA5/ERA5-TREFHT-ANN-global.png_actual.png and its corresponding box plot. I will try go through in details. |
|
@chengzhuzhang @tomvothecoder If you want to test using Workflow# Set up branch
cd <zppy directory>
git status
# Check that there are no file changes that will persist when we switch branches.
git fetch origin main
# Check out local version of my branch
# You need to do this because zppy can't run post-CDAT E3SM Diags with its code as of `main`.
git checkout -b issue-346-diags-post-refactor origin/issue-346-diags-post-refactor
# Set up E3SM Diags environment
cd <e3sm_diags directory>
git status
# Check that there are no file changes that will persist when we switch branches.
git fetch upstream
git checkout main
git reset --hard upstream/main
conda clean --all --y
conda env create -f conda-env/dev.yml -n e3sm_diags_main_<date>
conda activate e3sm_diags_main_<date>
pip install .
# Set up zppy environment
cd <zppy directory>
conda clean --all --y
conda env create -f conda/dev.yml -n zppy_dev_pr651
conda activate zppy_dev_pr651
pip install .If you want quick debugging method:For a sample test cfg, you could modify the paths in # Run zppy
zppy -c test.cfgIf you want to produce my test results:Full test reproduction steps# Set up a zppy-interfaces environment too
# We need to do this because the full test suite is expecting we have all tasks operating.
cd <zppy-interfaces>
git status
# Check that there are no file changes that will persist when we switch branches.
git fetch upstream
git checkout main
git reset --hard upstream/main
conda clean --all --y
conda env create -f conda-env/dev.yml -n zi_main_<date>
conda activate zi_main_<date>
pip install .
# Modify tests/integration/utils.py:
# UNIQUE_ID = "my-test-id"
# For get_chyrsalis_expansions: (Switch out the paths so they point to your conda install location)
# "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm_diags_main_<date>",
# "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi_main_<date>",
python tests/integration/utils.py
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 1st part of bundles cfg
# Once those all finish:
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg # Runs 2nd part of bundles cfg
# Once that finishes:
# Check output, grep lines should print nothing
cd /lcrc/group/e3sm/${USER}/zppy_weekly_comprehensive_v3_output/my-test-id/v3.LR.historical_0051/post/scripts/
grep -v "OK" *status
cd /lcrc/group/e3sm/${USER}/zppy_weekly_comprehensive_v2_output/my-test-id/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
cd /lcrc/group/e3sm/${USER}/zppy_weekly_bundles_output/my-test-id/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Run integration tests
cd <zppy directory>
pytest tests/integration/test_*.py
Now look for the Those have actual, expected, and diff images. |
|
thanks @forsyth2 ! |
|
I manually went through the diffs and noted some outstanding issues we need to address:
|
Yes, E3SM-Project/zppy#651 removes The only other issue with CDAT was with global-time-series, which was resolved in E3SM-Project/zppy#519 + E3SM-Project/zppy#611 (same situation -- the environment that In summary, the |
14 tasks
|
@tomvothecoder I can reproduce first issue (TREFHT) with a dev environment created from I'm wondering have you encountered similar problems before? |
Here is my comment in PR #903 with the regression testing notebook. The variables that you pointed out here with issues are also appearing in the list of mismatching variables. |
Hey @chengzhuzhang, I think I ran into this with I just created the dev environment and was able to mamba env create -f conda-env/dev.yml -n e3sm_diags_dev_894_2
mamba activate e3sm_diags_dev_894_2
(e3sm_diags_dev_894_2) vo13@login17:.../E3SM-Project/e3sm_diags$ python
Python 3.12.8 | packaged by conda-forge | (main, Dec 5 2024, 14:24:40) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import xcdat
>>>
KeyboardInterrupt
>>>
[1]+ Stopped python
(e3sm_diags_dev_894_2) vo13@login17:.../E3SM-Project/e3sm_diags$ conda list
# packages in environment at /global/homes/v/vo13/mambaforge/envs/e3sm_diags_dev_894_2:
#
# Name Version Build Channel
...
esmf 8.7.0 nompi_h6063b07_1 conda-forge
esmpy 8.7.0 pyhecae5ae_0 conda-forge
...
xarray 2024.11.0 pyhd8ed1ab_0 conda-forge
xcdat 0.7.3 pyhd8ed1ab_1 conda-forge
xesmf 0.8.8 pyhd8ed1ab_1 conda-forge |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened?
Related to E3SM-Project/zppy#651 (comment)
Image check failures
There's a large number of image check diffs in subdirectories of:
But as far as I can tell, the errors are mostly benign. Example errors:
actualhaving "RMSE" and "CORR" further to the left than inexpected.Some errors seem more concerning:
What did you expect to happen? Are there are possible answers you came across?
No response
Minimal Complete Verifiable Example (MVCE)
No response
Relevant log output
No response
Anything else we need to know?
No response
Environment
Latest
mainwith Xarray/xCDAT codebaseThe text was updated successfully, but these errors were encountered: