Marked as blocked:dependency because, at the moment, Externals.cfg points to branches of my personal MOSART (issue ESCOMP/MOSART#53) and RTM (issue ESCOMP/RTM#31) forks that contain the same changes. I haven't yet submitted those as pull requests, because I first wanted to see if any more testing is requested.

UPDATE
PRs ESCOMP/MOSART#66 and ESCOMP/RTM#35

From CTSM software engineering meeting today: This doesn't need to wait on other time-related PRs. It can come in as soon as the relevant externals are updated (and Externals.cfg here is changed to point to them).

derecho testing prior to updating .gitmodules to mosart1.1.08 and rtm1_0_85 in ce7f7b2:

PASS ./build-namelist_test.pl
PASS black and lint, though the latter gave 2 warnings
PASS ./run_ctsm_py_tests

Note to self: Consider initiating further testing, but perform final testing after updating to ctsm5.3.020.

Questions:
Given that we will mark this tag as the release tag for ctsm5.3:

• Would it make sense for us to include the one-line ccs_config update to enable f09_t232?
• Is it complicated to make a new ccs_config tag? I doubt that I have permission.
• Or is there a way around making a new ccs_config tag?
• Or should we wait for the fix to come in later?

@slevis-lmwg this is a good question. If the ccs_config updates can come in quickly, I'm supportive of wrapping the external update into this tag.

If this will take long, however, I'm wary of scope creep and somewhat less worried about how we're defining there minor version tags (especially since we expect a 5.4 tag to be made in the next few months). I'm also hesitant to hold up our upcoming tags, because it seemed like there are several FATES tags that are pretty critically and backed up against 5.3.021.

@slevis-lmwg this is a good question. If the ccs_config updates can come in quickly, I'm supportive of wrapping the external update into this tag.

If this will take long, however, I'm wary of scope creep and somewhat less worried about how we're defining there minor version tags (especially since we expect a 5.4 tag to be made in the next few months). I'm also hesitant to hold up our upcoming tags, because it seemed like there are several FATES tags that are pretty critically and backed up against 5.3.021.

Yay, I see that this was addressed here:
ESMCI/ccs_config_cesm#205
Thank you @fischer-ncar and @wwieder

Nice that we have a tag for the simple fix needed. So you should just include that here.

I will say that I'm supportive of this NOT having everything clean even for this tag being a development release tag. At least in this case when running with MOM masks in stand-alone CTSM is still considered "experimental". I know it shouldn't be thought of that way, but until it's standard practice -- it's experimental. The fact that it was broken the first time we tried it -- just underscores how "experimental" it really is right now.

And @slevis-lmwg just to be explicit on your questions:

• Is it complicated to make a new ccs_config tag? I doubt that I have permission.

It's not complicated, once a ccs_config PR is merged it automatically creates a tag for it.

BUT what is critical here is to have done enough testing with it that it doesn't break something in CESM. In this case it was easy to approve because it was so isolated to handling a grid that wasn't working. So your testing was sufficient.

@jedwards4b and @fischer-ncar are the main contributors to ccs_config from CSEG. There's a few others too though.

• Or is there a way around making a new ccs_config tag?

Not, really other than maybe doing a branch tag, but I don't think we should do that most of the time if ever.

Also there's a possible caveat to updating to the latest ccs_config:
ctsm5.3.020 is updating to ccs_config_cesm1.0.16. Beyond that I see tags upto ccs_config_cesm1.0.19, so the f09_t232 fix is not the only thing that we'd be getting by updating...

It looks like those changes in ccs_config tags should be benign. But, they also haven't gone through CESM testing, so I'd treat them a little suspect. I suggest you try it with it updated, but if it causes any problems -- back that update out.

derecho:

PASS ./build-namelist_test.pl
PASS black and lint, though the latter gave 2 warnings
PASS ./run_ctsm_py_tests
PASS ./run_sys_tests -s rtm -c rtm1_0_86-ctsm5.3.019 -g rtm1_0_86-ctsm5.3.021
OK ./run_sys_tests -s mosart -c mosart1.1.08-ctsm5.3.019 -g mosart1.1.08-ctsm5.3.021
DIFF ./run_sys_tests -s ctsm_sci -c ctsm_sci-ctsm5.3.016 -g ctsm_sci-ctsm5.3.021
DIFF ./run_sys_tests -s fates -c fates-sci.1.80.4_api.37.0.0-tmp-241219.n02.ctsm5.3.016 -g fates-sci.1.80.4_api.37.0.0-ctsm5.3.021
FAIL ./run_sys_tests -s aux_clm -c ctsm5.3.020 -g ctsm5.3.021

ctsm_sci tests_0117-125909de ./cs.status.fails | grep -v '16: DI' | grep -v NLCOMP

FAIL RXCROPMATURITY_Lm61.f09_g17.IHistClm60BgcCrop.derecho_intel.clm-cropMonthOutput RUN

aux_clm tests_0117-125438de ./cs.status.fails | grep -v PASS | grep -v 'ise bit-for'

FAIL ERS_Ly3_P64x2.f10_f10_mg37.IHistClm50BgcCropG.derecho_intel.clm-cropMonthOutput DIFF mosart/cpl expected
FAIL RXCROPMATURITYSKIPGEN_Ld1097.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-cropMonthOutput RUN
FAIL SMS_Ly3.f10_f10_mg37.I1850Clm50SpG.derecho_intel.clm-glcMEC_long--clm-nofireemis DIFF mosart/cpl expected

izumi

PASS ./run_sys_tests -s mosart -c mosart1.1.08-ctsm5.3.019 -g mosart1.1.08-ctsm5.3.021
DIFF ./run_sys_tests -s fates -c fates-sci.1.80.4_api.37.0.0-ctsm5.3.014 -g fates-sci.1.80.4_api.37.0.0-ctsm5.3.021
FAIL ./run_sys_tests -s aux_clm -c ctsm5.3.020 -g ctsm5.3.021

fates tests_0117-131357iz

• FAIL ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.izumi_nag.clm-FatesColdPRT2 CREATE_NEWCASE
  Submitted again but with ./create_test and ran out of wallclock, so resubmitted and ran out of wallclock again, so tried ./create_test ... --wallclock 04:00:00 and ran out of wallclock a third time, so now increase the test's wallclock in testlist_clm.xml BUT FAILED AGAIN. The email notification always says that walltime was 1 hour.

aux_clm tests_0117-131226iz

• FAIL ERS_D_Ld20.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream CREATE_NEWCASE
  Submitted again but with ./create_test and got (check whether it's expected):

FAIL ERS_D_Ld20.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesColdTwoStream COMPARE_base_rest

I rushed to start testing and didn't first update to ccs_config_cesm1.0.20.
But... maybe that's better, so as to confirm first that everything works without that update.

@samsrabin I'm afraid the RXCROP... test fails again in the latest aux_clm. You can look here:
/glade/derecho/scratch/slevis/tests_0117-125438de/RXCROPMATURITYSKIPGEN_Ld1097.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-cropMonthOutput.GC.0117-125438de_int

Did we miss bringing in your changes or something?

UPDATE
I interpret git diff 9b71518 to suggest that we didn't miss anything, and that it's all there as it was before the latest updates. I think this means that the RXCROP... tests fail due to something new in the updates. @samsrabin I'm coming up with two options:

1. "time" is now the middle of "time_bounds"
2. the elimination of the 0th time step

I thought that I had tested your latest version, but from looking again it seems as though I had not. I'm happy to help troubleshoot this next week if you like.

The error that I see:

../../../python/ctsm/crop_calendars/cropcal_module.py", line 40, in check_and_trim_years
    f"Expected {n_years_expected} timesteps in output but got {ds_in.dims['time']}"
RuntimeError: Expected 2 timesteps in output but got 1