Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature]: Adapt CDAT Migration regression testing scripts for general purpose use #903

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from
Draft

[Feature]: Adapt CDAT Migration regression testing scripts for general purpose use #903

wants to merge 8 commits into from
+3,938 −10

Conversation

tomvothecoder
Copy link
Collaborator

@tomvothecoder tomvothecoder commented Dec 6, 2024 •
edited
Loading

Description

Todos

Adapt regression testing scripts

  • Adapt the comparison notebook to a Python script and automate -- IN PROGRESS

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • My changes generate no new warnings
  • Any dependent changes have been merged and published in downstream modules

If applicable:

  • New and existing unit tests pass with my changes (locally and CI/CD build)
  • I have added tests that prove my fix is effective or that my feature works
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have noted that this is a breaking change for a major release (fix or feature that would cause existing functionality to not work as expected)

@tomvothecoder tomvothecoder mentioned this pull request Dec 6, 2024
Closed
9 tasks
@chengzhuzhang
Copy link
Contributor

@tomvothecoder thank you for working on this. I'm considering to have both image checker and the new regression tests based on nc files and json. For cases that .nc files are generated correctly, but images are not. We can talk about more on Monday.

P.S. congratulations on the big merge and a brand new code base!

@tomvothecoder
Copy link
Collaborator Author

Since this regression testing will be run on HPC and not GH Actions, we won't need to containerize the data or make the image checker an optional test.

@tomvothecoder
Copy link
Collaborator Author

@chengzhuzhang Once this is merged, should we do a full run with v3 data on latest main? Might be a good idea before releasing v3.0.0.

@chengzhuzhang
Copy link
Contributor

@chengzhuzhang Once this is merged, should we do a full run with v3 data on latest main? Might be a good idea before releasing v3.0.0.

Absolutely. When releasing v2.12.1, I only did metrics and image checking because the netcdf files were not saved in the full run. We should save the nc files this time..

@tomvothecoder
Copy link
Collaborator Author

tomvothecoder commented Dec 9, 2024 •
edited
Loading

@chengzhuzhang Once this is merged, should we do a full run with v3 data on latest main? Might be a good idea before releasing v3.0.0.

Absolutely. When releasing v2.12.1, I only did metrics and image checking because the netcdf files were not saved in the full run. We should save the nc files this time..

@chengzhuzhang Actually we do need to run the latest main with v3 data for this PR. That way I can write the integration tests and try them on perlmutter.

Can you provide the specific path to where I should save the results for persistence?

EDIT: I made this dir to store results: /global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main

@chengzhuzhang
Copy link
Contributor

EDIT: I made this dir to store results: /global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main

This is good! Thank you.

@tomvothecoder tomvothecoder force-pushed the tests/894-regression-test branch from 880b6fe to 5831834 Compare December 10, 2024 22:03
@tomvothecoder tomvothecoder marked this pull request as ready for review December 10, 2024 22:03
@tomvothecoder tomvothecoder marked this pull request as draft December 10, 2024 22:03
@tomvothecoder
Copy link
Collaborator Author

Hey @chengzhuzhang I want to perform a regression test with v3 data using v2.12.1 and the latest main. I'm trying to re-run the v3 script with v2.12.1 and save_netcdf=True to save outputs to netCDF.

However, I keep getting concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending. at some point. Any ideas why?

Commands

mamba create -n e3sm_diags_2.12.1 -c conda-forge e3sm_diags=2.12.1
mamba activate e3sm_diags_2.12.1

python auxiliary_tools/complete_run_script.py

Script

"""
This script sets up and runs a series of diagnostics for the E3SM model output.

The diagnostics include:
- ENSO diagnostics
- Tropical subseasonal variability diagnostics
- QBO diagnostics
- Diurnal cycle diagnostics
- Streamflow diagnostics
- Tropical cyclone analysis
- ARM diagnostics

The script configures the parameters for each diagnostic, including paths to
model output and observational data, time periods for analysis, and output
settings. It then runs the diagnostics using the e3sm_diags package.

Parameters:
- case: The name of the model case.
- short_name: A short name for the model case.
- results_dir: Directory where the results will be saved.
- test_climo: Path to the model climatology data.
- test_ts: Path to the model time-series data.
- test_ts_daily_dir: Path to the model daily time-series data.
- ref_climo: Path to the reference climatology data.
- ref_ts: Path to the reference time-series data.
- start_yr: Start year for the analysis.
- end_yr: End year for the analysis.

The script uses multiprocessing to speed up the diagnostics computation.

Example usage:
    python complete_run_script.py
"""

from e3sm_diags.parameter.arm_diags_parameter import ARMDiagsParameter
from e3sm_diags.parameter.core_parameter import CoreParameter
from e3sm_diags.parameter.diurnal_cycle_parameter import DiurnalCycleParameter
from e3sm_diags.parameter.enso_diags_parameter import EnsoDiagsParameter
from e3sm_diags.parameter.qbo_parameter import QboParameter
from e3sm_diags.parameter.streamflow_parameter import StreamflowParameter
from e3sm_diags.parameter.tc_analysis_parameter import TCAnalysisParameter
from e3sm_diags.parameter.tropical_subseasonal_parameter import (
    TropicalSubseasonalParameter,
)
from e3sm_diags.run import runner

case = "extendedOutput.v3.LR.historical_0101"
short_name = "v3.LR.historical_0101"

# TODO: Update `RESULTS_DIRNAME` as needed.
RESULTS_DIRNAME = "v2.12.1v2"
results_dir = f"/global/cfs/cdirs/e3sm/www/e3sm_diags/{RESULTS_DIRNAME}/"

test_climo = "/global/cfs/cdirs/e3sm/chengzhu/tutorial2024/v3.LR.historical_0101/post/atm/180x360_aave/clim/15yr"
test_ts = "/global/cfs/cdirs/e3sm/chengzhu/tutorial2024/v3.LR.historical_0101/post/atm/180x360_aave/ts/monthly/15yr"
test_ts_daily_dir = "/global/cfs/cdirs/e3sm/chengzhu/tutorial2024/v3.LR.historical_0101/post/atm/180x360_aave/ts/daily/15yr"

ref_climo = "/global/cfs/cdirs/e3sm/diagnostics/observations/Atm/climatology/"
ref_ts = "/global/cfs/cdirs/e3sm/diagnostics/observations/Atm/time-series"

start_yr = "2000"
end_yr = "2014"

param = CoreParameter()

# Model
param.test_data_path = test_climo
param.test_name = case
param.short_test_name = short_name

# Ref/Obs
param.reference_data_path = ref_climo

# Output dir
param.results_dir = results_dir

# Additional settings
param.run_type = "model_vs_obs"
param.diff_title = "Model - Observations"
param.output_format = ["png"]
param.output_format_subplot = []
param.multiprocessing = True
param.num_workers = 24
param.save_netcdf = True
param.seasons = ["ANN"]
params = [param]

# Model
enso_param = EnsoDiagsParameter()
enso_param.test_data_path = test_ts
# enso_param.test_name = short_name
enso_param.test_start_yr = start_yr
enso_param.test_end_yr = end_yr

# Obs
enso_param.reference_data_path = ref_ts
enso_param.ref_start_yr = start_yr
enso_param.ref_end_yr = end_yr

enso_param.save_netcdf = True
params.append(enso_param)

trop_param = TropicalSubseasonalParameter()
trop_param.test_data_path = test_ts_daily_dir
# trop_param.test_name = short_name
trop_param.test_start_yr = start_yr
trop_param.test_end_yr = end_yr

# Obs
trop_param.reference_data_path = ref_ts
trop_param.ref_start_yr = "2001"
trop_param.ref_end_yr = "2010"

trop_param.save_netcdf = True
params.append(trop_param)

qbo_param = QboParameter()
qbo_param.test_data_path = test_ts
# qbo_param.test_name = short_name
qbo_param.test_start_yr = start_yr
qbo_param.test_end_yr = end_yr
qbo_param.ref_start_yr = start_yr
qbo_param.ref_end_yr = end_yr

# Obs
qbo_param.reference_data_path = ref_ts

qbo_param.save_netcdf = True
params.append(qbo_param)

dc_param = DiurnalCycleParameter()
dc_param.test_data_path = "/global/cfs/cdirs/e3sm/chengzhu/tutorial2024/v3.LR.historical_0101/post/atm/180x360_aave/clim_diurnal_8xdaily/"
# dc_param.short_test_name = short_name
# Plotting diurnal cycle amplitude on different scales. Default is True
dc_param.normalize_test_amp = False

# Obs
dc_param.reference_data_path = ref_climo

dc_param.save_netcdf = True
params.append(dc_param)

streamflow_param = StreamflowParameter()
streamflow_param.reference_data_path = ref_ts
streamflow_param.test_data_path = "/global/cfs/cdirs/e3sm/chengzhu/tutorial2024/v3.LR.historical_0101/post/rof/native/ts/monthly/15yr/"
# streamflow_param.test_name = short_name
streamflow_param.test_start_yr = start_yr
streamflow_param.test_end_yr = end_yr

# Obs
streamflow_param.reference_data_path = ref_ts
streamflow_param.ref_start_yr = (
    "1986"  # Streamflow gauge station data range from year 1986 to 1995
)
streamflow_param.ref_end_yr = "1995"

streamflow_param.save_netcdf = True
params.append(streamflow_param)

tc_param = TCAnalysisParameter()
tc_param.test_data_path = "/global/cfs/cdirs/e3sm/chengzhu/tutorial2024/v3.LR.historical_0101/post/atm/tc-analysis_2000_2014"
# tc_param.short_test_name = short_name
tc_param.test_start_yr = start_yr
tc_param.test_end_yr = end_yr

# Obs
tc_param.reference_data_path = (
    "/global/cfs/cdirs/e3sm/diagnostics/observations/Atm/tc-analysis/"
)
# For model vs obs, the ref start and end year can be any four digit strings
# For now, use all available years from obs by default
tc_param.ref_start_yr = "1979"
tc_param.ref_end_yr = "2018"

tc_param.save_netcdf = True
params.append(tc_param)

arm_param = ARMDiagsParameter()
arm_param.reference_data_path = (
    "/global/cfs/cdirs/e3sm/diagnostics/observations/Atm/arm-diags-data"
)
arm_param.ref_name = "armdiags"
arm_param.test_data_path = (
    "/global/cfs/cdirs/e3sm/chengzhu/tutorial2024/v3.LR.historical_0101/post/atm/site"
)
arm_param.test_name = short_name
arm_param.test_start_yr = start_yr
arm_param.test_end_yr = end_yr
# For model vs obs, the ref start and end year can be any four digit strings.
# For now, will use all available years form obs
arm_param.ref_start_yr = "0001"
arm_param.ref_end_yr = "0001"

arm_param.save_netcdf = True
params.append(arm_param)

# Run
runner.sets_to_run = [
    "lat_lon",
    "zonal_mean_xy",
    "zonal_mean_2d",
    "polar",
    "cosp_histogram",
    "meridional_mean_2d",
    "enso_diags",
    "qbo",
    "diurnal_cycle",
    "annual_cycle_zonal_mean",
    "streamflow",
    "zonal_mean_2d_stratosphere",
    "arm_diags",
    "tc_analysis",
    "aerosol_aeronet",
    "aerosol_budget",
    "tropical_subseasonal",
]

runner.run_diags(params)

Partial Traceback

2024-12-10 16:29:58,256 [ERROR]: run.py(run_diags:91) >> Error traceback:
Traceback (most recent call last):
  File "/global/homes/v/vo13/mambaforge/envs/e3sm_diags_2.12.1/lib/python3.10/site-packages/e3sm_diags/run.py", line 89, in run_diags
    params_results = main(params)
  File "/global/homes/v/vo13/mambaforge/envs/e3sm_diags_2.12.1/lib/python3.10/site-packages/e3sm_diags/e3sm_diags_driver.py", line 371, in main
    parameters_results = _run_with_dask(parameters)
  File "/global/homes/v/vo13/mambaforge/envs/e3sm_diags_2.12.1/lib/python3.10/site-packages/e3sm_diags/e3sm_diags_driver.py", line 316, in _run_with_dask
    results = bag.map(CoreParameter._run_diag).compute(num_workers=num_workers)
  File "/global/homes/v/vo13/mambaforge/envs/e3sm_diags_2.12.1/lib/python3.10/site-packages/dask/base.py", line 372, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/global/homes/v/vo13/mambaforge/envs/e3sm_diags_2.12.1/lib/python3.10/site-packages/dask/base.py", line 660, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/global/homes/v/vo13/mambaforge/envs/e3sm_diags_2.12.1/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
2024-12-10 16:29:58,267 [INFO]: logger.py(move_log_to_prov_dir:106) >> Log file saved in /global/cfs/cdirs/e3sm/www/e3sm_diags/v2.12.1v2//prov/e3sm_diags_run.log

@chengzhuzhang
Copy link
Contributor

This is coming back haunting us... I haven't been seen this problem for a while though.

@xylar
Copy link
Contributor

xylar commented Dec 11, 2024

In #720, it turned out that the import order was fragile. I don't think we really ever figured out why but it may be something in esmpy that's not implemented like it should be. In working with the ESMF developers, I get the sense that python experience is lacking there.

How is esmpy still used in e3sm_diags now that cdat has been removed? Through xcdat?

@tomvothecoder
Copy link
Collaborator Author

This is coming back haunting us... I haven't been seen this problem for a while though.

@chengzhuzhang Can you try to reproduce it on your end with the commands in my comment above? I wonder if I'm doing something incorrect, although I'm confident I'm not.

How is esmpy still used in e3sm_diags now that cdat has been removed? Through xcdat?

@xylar The latest main branch uses esmpy via xcdat + xesmf.

This reminds me to remove esmpy from the dependencies since it is not imported directly.

@chengzhuzhang
Copy link
Contributor

chengzhuzhang commented Dec 11, 2024 •
edited
Loading

@chengzhuzhang Can you try to reproduce it on your end with the commands in my comment above? I wonder if I'm doing something incorrect, although I'm confident I'm not.

Okay, let me try reproduce, following these steps. @tomvothecoder are you running on login node?

@chengzhuzhang
Copy link
Contributor

@tomvothecoder I suspect that the concurrent.futures.process.BrokenProcessPool is caused by memory constrain if running on login node. For Perlmutter login node, it has 54 GB limit. I had the same issue when I run the complete run on login, but the the same run is going fine on a compute node. I used following to get an interactive node and run: salloc --nodes 1 --qos interactive --time 02:00:00 --constraint cpu --account e3sm

@tomvothecoder
Various updates to tests related files
b732f14 
- Add `clean-test-integration` command to `Makefile` and delete `clean_tests_repo.sh`
- Move params for `complete_run.py` to `complete_run_params.py` for re-use in `test_complete_run.py`
- Rename `tests/integration/complete_run.py` to `old_complete_run.py`
@chengzhuzhang
Copy link
Contributor

@tomvothecoder the run just completed here: /global/cfs/cdirs/e3sm/www/chengzhu/e3sm_diags/v2.12.1v2/
I don't have permission for /global/cfs/cdirs/e3sm/www/e3sm_diags, feel free to make a copy to same time..

@tomvothecoder
Copy link
Collaborator Author

@tomvothecoder I suspect that the concurrent.futures.process.BrokenProcessPool is caused by memory constrain if running on login node. For Perlmutter login node, it has 54 GB limit. I had the same issue when I run the complete run on login, but the the same run is going fine on a compute node. I used following to get an interactive node and run: salloc --nodes 1 --qos interactive --time 02:00:00 --constraint cpu --account e3sm

Yes, I am running on the login node. As you suggest, maybe the CDAT codebase is constrained by the 54 GB memory limit.

The good news is that the script runs fine with the Xarray/xCDAT codebase, which suggests the new backend is more memory efficient.

@tomvothecoder the run just completed here: /global/cfs/cdirs/e3sm/www/chengzhu/e3sm_diags/v2.12.1v2/ I don't have permission for /global/cfs/cdirs/e3sm/www/e3sm_diags, feel free to make a copy to same time..

Thank you! I will copy the results over.

@chengzhuzhang
Copy link
Contributor

The good news is that the script runs fine with the Xarray/xCDAT codebase, which suggests the new backend is more memory efficient.

Nice to know!

@tomvothecoder tomvothecoder changed the title [Feature]: Adapt CDAT Migration regression testing scripts as integration tests to replace image checker [Feature]: Adapt CDAT Migration regression testing scripts for general purpose use Dec 11, 2024
@tomvothecoder
Copy link
Collaborator Author

tomvothecoder commented Dec 11, 2024 •
edited
Loading

Here is the regression notebook that compares latest main vs. v2.12.1.

  • Matching number of .nc files produced (590)
  • 501/590 are matching
  • 4/590 are mismatching (nan positions)
  • 85/590 are not equal

For the 85/590 are not equal to rtol=1e-4. These variables will need further investigation.

['/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/COREv2_Flux/COREv2_Flux-PminusE-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/CRU_IPCC/CRU-TREFHT-ANN-land_60S90N_ref.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/CRU_IPCC/CRU-TREFHT-ANN-land_60S90N_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/Cloud MISR/MISRCOSP-CLDTOT_TAU1.3_9.4_MISR-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/Cloud MISR/MISRCOSP-CLDTOT_TAU1.3_MISR-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/ERA5/ERA5-NET_FLUX_SRF-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/ERA5/ERA5-OMEGA-200-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/ERA5/ERA5-OMEGA-500-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/ERA5/ERA5-OMEGA-850-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/ERA5/ERA5-TREFHT-ANN-global_ref.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/ERA5/ERA5-TREFHT-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/ERA5/ERA5-TREFHT-ANN-land_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/ERA5/ERA5-U-850-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/GPCP_OAFLux/GPCP_OAFLux-PminusE-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/MERRA2/MERRA2-NET_FLUX_SRF-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/MERRA2/MERRA2-OMEGA-200-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/MERRA2/MERRA2-OMEGA-500-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/MERRA2/MERRA2-OMEGA-850-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/MERRA2/MERRA2-TREFHT-ANN-global_ref.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/MERRA2/MERRA2-TREFHT-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/MERRA2/MERRA2-TREFHT-ANN-land_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/MERRA2/MERRA2-TREFMNAV-ANN-global_ref.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/MERRA2/MERRA2-TREFMNAV-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/MERRA2/MERRA2-TREFMXAV-ANN-global_ref.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/MERRA2/MERRA2-TREFMXAV-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/SST_CL_HadISST/HadISST_CL-SST-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/SST_HadISST/HadISST-SST-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/SST_PD_HadISST/HadISST_PD-SST-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/lat_lon/SST_PI_HadISST/HadISST_PI-SST-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/meridional_mean_2d/MERRA2/MERRA2-OMEGA-ANN-global_ref.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/CRU_IPCC/CRU-TREFHT-ANN-polar_N_ref.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/CRU_IPCC/CRU-TREFHT-ANN-polar_N_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/ERA5/ERA5-TREFHT-ANN-polar_N_ref.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/ERA5/ERA5-TREFHT-ANN-polar_N_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/ERA5/ERA5-TREFHT-ANN-polar_S_ref.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/ERA5/ERA5-TREFHT-ANN-polar_S_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/ERA5/ERA5-U-850-ANN-polar_N_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/ERA5/ERA5-U-850-ANN-polar_S_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/MERRA2/MERRA2-TREFHT-ANN-polar_N_ref.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/MERRA2/MERRA2-TREFHT-ANN-polar_N_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/MERRA2/MERRA2-TREFHT-ANN-polar_S_ref.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/MERRA2/MERRA2-TREFHT-ANN-polar_S_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/MERRA2/MERRA2-TREFMNAV-ANN-polar_N_ref.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/MERRA2/MERRA2-TREFMNAV-ANN-polar_N_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/MERRA2/MERRA2-TREFMNAV-ANN-polar_S_ref.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/MERRA2/MERRA2-TREFMNAV-ANN-polar_S_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/MERRA2/MERRA2-TREFMXAV-ANN-polar_N_ref.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/MERRA2/MERRA2-TREFMXAV-ANN-polar_N_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/MERRA2/MERRA2-TREFMXAV-ANN-polar_S_ref.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/MERRA2/MERRA2-TREFMXAV-ANN-polar_S_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/SST_CL_HadISST/HadISST_CL-SST-ANN-polar_N_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/SST_CL_HadISST/HadISST_CL-SST-ANN-polar_S_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/SST_PD_HadISST/HadISST_PD-SST-ANN-polar_N_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/SST_PD_HadISST/HadISST_PD-SST-ANN-polar_S_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/SST_PI_HadISST/HadISST_PI-SST-ANN-polar_N_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/polar/SST_PI_HadISST/HadISST_PI-SST-ANN-polar_S_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/zonal_mean_2d/ERA5/ERA5-OMEGA-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/zonal_mean_2d/MERRA2/MERRA2-OMEGA-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/zonal_mean_2d_stratosphere/ERA5/ERA5-H2OLNZ-ANN-global_ref.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/zonal_mean_2d_stratosphere/ERA5/ERA5-H2OLNZ-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/zonal_mean_2d_stratosphere/MERRA2/MERRA2-H2OLNZ-ANN-global_ref.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/zonal_mean_2d_stratosphere/MERRA2/MERRA2-H2OLNZ-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/zonal_mean_xy/ERA5/ERA5-TREFHT-ANN-global_ref.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/zonal_mean_xy/ERA5/ERA5-TREFHT-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/zonal_mean_xy/MERRA2/MERRA2-TREFHT-ANN-global_ref.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/zonal_mean_xy/MERRA2/MERRA2-TREFHT-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/zonal_mean_xy/MERRA2/MERRA2-TREFMNAV-ANN-global_ref.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/zonal_mean_xy/MERRA2/MERRA2-TREFMNAV-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/zonal_mean_xy/MERRA2/MERRA2-TREFMXAV-ANN-global_ref.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/zonal_mean_xy/MERRA2/MERRA2-TREFMXAV-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/zonal_mean_xy/SST_CL_HadISST/HadISST_CL-SST-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/zonal_mean_xy/SST_PD_HadISST/HadISST_PD-SST-ANN-global_test.nc',
 '/global/cfs/cdirs/e3sm/www/e3sm_diags/complete_run/24-12-09-main/zonal_mean_xy/SST_PI_HadISST/HadISST_PI-SST-ANN-global_test.nc']

This was referenced Dec 11, 2024
Open
Draft
@xylar
Copy link
Contributor

xylar commented Dec 12, 2024

Maybe NERSC policies have changed but I have got my account suspended in the past for doing heavy computing on login nodes. I'd strongly recommend interactive jobs on compte nodes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
FY25 Q1 (10/1/24 - 12/31/24)
3 participants
@tomvothecoder @chengzhuzhang @xylar