2-deg I-case spinup and historical baseline for PPE with BNF tag (ctsm53017_f19_BNF_[AD, SASU, pSASU, hist])

[linniahawkins]

Description:
For the CLM6-PPE, we need a default 2-deg full grid simulation for validation. Yes- you already ran one of these for me but we switched to a new tag and are using a different default parameter set now.

Tag: https://github.com/ESCOMP/CTSM/releases/tag/branch_tags%2FBNF_v2.n01_ctsm5.3.012
Parameter file: /glade/work/linnia/CLM6-PPE/ctsm6_lhc/paramfiles/lhc0000.nc
Forcing data: CRUJRA
do_grossunrep = .false.

Let me know if I missed anything, Thanks!

---

Case directory:

./create_newcase --case ~slevis/cases_LMWG_dev/ctsm53017_f19_BNF_AD --compset 1850_DATM%CRUv7_CLM60%BGC_SICE_SOCN_MOSART_SGLC_SWAV --res f19_g17 --run-unsupported
./create_clone --clone ~slevis/cases_LMWG_dev/ctsm53017_f19_BNF_AD/ --case ~slevis/cases_LMWG_dev/ctsm53017_f19_BNF_SASU
./create_newcase --case ~slevis/cases_LMWG_dev/ctsm53017_f19_BNF_hist --compset HIST_DATM%CRUv7_CLM60%BGC_SICE_SOCN_MOSART_SGLC_SWAV --res f19_g17 --run-unsupported

---

Sandbox:
Locally:
/glade/work/slevis/git/LMWG_dev8
git branch: tmp-241219 (temporary branch while master is "locked" in preparation for a cesm tag)
git describe: branch_tags/tmp-241219.n01.ctsm5.3.016 (would have been ctsm5.3.017 if we were not keeping the master branch "locked")
In #88 (here) and #89 I had to update cime6.1.49 to cime6.1.56 to avoid the rpointer.cpl bug at restart...

---

user_nl_ changes:
Relative to #72

20,22c20,22
< nfix_method = 'Bytnerowicz'
< paramfile = '/glade/work/linnia/CLM6-PPE/ctsm6_lhc/paramfiles/lhc0000.nc'
< fsurdat = '/glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/surfdata_esmf/ctsm5.3.0/surfdata_1.9x2.5_hist_1850_16pfts_c240926.nc'
---
> paramfile = '/glade/work/wwieder/modified_inputs/ctsm60_params_nfix.c241101.nc'
> ! The 78pft version comes out of the box with this setup, so point explicitly to a 16pft version
> fsurdat = '/glade/derecho/scratch/slevis/temp_work/new_rawdata/tools/mksurfdata_esmf/surfdata_1.9x2.5_hist_1850_16pfts_c240926.nc'

Additional diffs to start the historical segment of the spin-up:

< flanduse_timeseries = '/glade/campaign/cesm/cesmdata/inputdata/lnd/clm2/surfdata_esmf/ctsm5.3.0/landuse.timeseries_1.9x2.5_hist_1850-2023_16pfts_c240926.nc'
< use_init_interp = .true.
< finidat = '/glade/derecho/scratch/slevis/archive/ctsm53017_f19_BNF_pSASU/rest/0201-01-01-00000/ctsm53017_f19_BNF_pSASU.clm2.r.0201-01-01-00000.nc'

---

SourceMods:
None

---

Diagnostics:
https://webext.cgd.ucar.edu/I20TR/ctsm53017_f19_BNF_hist/lnd/ctsm53017_f19_BNF_hist.2004_2023-ctsm530_f19_nfix_hist2_2004_2023/setsIndex.html

---

Output:
Sample paths shown for AD, but similar for SASU, etc.
Initially
/glade/derecho/scratch/slevis/ctsm53017_f19_BNF_AD/run
Later
/glade/derecho/scratch/slevis/archive/ctsm53017_f19_BNF_AD
In the end if we decide to keep longterm
/glade/campaign/cgd/tss/people/slevis/i_cases/cases_LMWG_dev/ctsm53017_f19_BNF_AD

---

Contacts:
@slevis-lmwg

---

Extra details:
AD: 300 yrs, 2.52% disequilibrium
SASU: 300 yrs, 12.45% disequilibrium
pSASU: 200 yrs, 1.29% disequilibrium
hist: 1850-2023

[wwieder]

Let's start this after we have b4b merged this week, @slevis-lmwg and then turn on the bytnerovich (spelling) function with a user_nl_clm change. Effectively this would be identical to #72, but with the latest tag and the latest parameter file used in #86 (/glade/work/linnia/CLM6-PPE/ctsm6_lhc/paramfiles/lhc0000.nc)

[linniahawkins]

Sounds good - but I just want to note that the paramfile in #86 is not the same as: /glade/work/linnia/CLM6-PPE/ctsm6_lhc/paramfiles/lhc0000.nc

But we can use the paramfile from #86 and I will run a sparsegrid simulation with that paramfile and use it for PPE validation. (this option saves us a fullgrid spinup).

Revised paramfile requested:
paramfile = '/glade/u/home/linnia/ctsm6_ppe/gen_ensembles/jobscripts/test/gen_paramfiles/ctsm5.3.012.Nfix_params.v5.c241212.nc'

[wwieder]

Sorry for the confusion. Are you sure you want the ctsm5.3.012.Nfix_params.v5.c241212.nc from #86 here, Linnia?

[linniahawkins]

Actually - it might be cleaner to have a full grid simulation with the lhc0000.nc paramfile when I am writing this up.

Sorry for waffling. Let's use the original paramfile requested:
/glade/work/linnia/CLM6-PPE/ctsm6_lhc/paramfiles/lhc0000.nc

[wwieder]

I like waffles, we'll stick with your lhc0000 parameter file at 2 degrees? Just confirming you want the full AD, SASU, postSASU and historical through 2023 here?

[linniahawkins]

Correct, thank you!

[slevis-lmwg]

b4b-dev took longer than expected due to last-minute revisions requested in a code review. It's ready now and awaiting approval in ESCOMP/CTSM#2917. This may take a while due to the holidays. I'm assuming that we want this simulation associated with a formal tag. If not, I could start the simulation right away, so let me know.

[slevis-lmwg]

@wwieder @linniahawkins I pointed to Will's TRENDY2024 directory on /scratch, as was done in #72 but did we make a safer / more permanent copy of this data somewhere? (I do not see it in /glade/campaign/cesm/cesmdata/inputdata/atm/datm7)

[wwieder]

No, we need to add TRENDY2024 to input data. I think there's a CTSM issue on this?

[slevis-lmwg]

I will look for the issue and make a copy of the data in /inputdata today before the scrubber beats us to it :-)

Meanwhile though the simulation has stopped after 80 yrs and fails to continue. It gives the
ERROR: CONTINUE_RUN is true but this case does not appear to have restart files staged in /glade/derecho/scratch/slevis/ctsm53017_f19_BNF_AD/run rpointer.cpl
For now this looks like an rpointer bug to me, because the files are right there, as far as I can tell. I will try to work around it and then I will consider opening an issue.

UPDATE1: Got past the previous error with
mv rpointer.cpl.0081-01-01-00000 rpointer.cpl
but the run now fails while looking for rpointer.cpl.0081-01-01-00000 :-(
UPDATE2: Got past both errors with
cp rpointer.cpl.0081-01-01-00000 rpointer.cpl :-)

[slevis-lmwg]

Today:

• Running AD for a few more decades hoping to get AD disequilibrium < 3%.
• Setting up SASU with cime6.1.56 (instead of cime6.1.49) as recommended in rpointer.cpl bug causes ctsm restart to fail ESCOMP/CTSM#2920 (comment), hoping to fix the rpointer bug discussed in that issue and above. UPDATE: this seems fixed, so i will update the issue.

[slevis-lmwg]

The SASU portion of this spin-up is taking longer than typical:

[slevis-lmwg]

For reference, the corresponding AD plots:

And for the record, in SASU I did NOT reseed_dead_plants. I know we've discussed this issue before. I thought reseeding should not be necessary after AD. But maybe I should try a second simulation in case this explains the slower spin-up...

[olyson]

The SASU spinup seems reasonable when compared to the last PPE spinup, #70? That was at 13.73% after 280 years.

[slevis-lmwg]

@olyson thank you for pointing that out.

In the mean time I extended SASU to year 600 (8.12% disequilibrium) and SASUreseed to year 400 (10.49% disequilibrium).

To keep from changing the spin-up approach, I will start pSASU from SASU year 300, but let me know if you prefer something different.

[slevis-lmwg]

Also I should note that our izumi nag tests are failing as explained here. Currently I suspect that a bug came in with the Bytnerowicz PR ESCOMP/CTSM#2869.

UPDATE: Fix for this bug (ESCOMP/CTSM#2924) may cause small change in answers due to possible shift by one of the nfix_topt, nfix_tmin, nfix_tmax arrays.

[olyson]

@olyson thank you for pointing that out.

In the mean time I extended SASU to year 600 (8.12% disequilibrium) and SASUreseed to year 400 (10.49% disequilibrium).

To keep from changing the spin-up approach, I will start pSASU from SASU year 300, but let me know if you prefer something different.

Sounds good to me.

[slevis-lmwg]

Just started the historical segment of this spin-up.

[wwieder]

@slevis-lmwg at some point can you turn on the h1 history output for this case, as in #70 or #72? I'd like to look at PFT survival with this new parameter set.

[slevis-lmwg]

@wwieder thank you for the reminder. As I see that I have done previously, I am adding to user_nl_clm:

ndep_taxmode = 'extend'

 hist_dov2xy(1) = .true.
 hist_dov2xy(2) = .false.
 hist_dov2xy(3) = .false.
 hist_dov2xy(4) = .true.
 hist_dov2xy(5) = .false.
 hist_fields_list_file = .false.
 hist_fincl2 = 'TLAI', 'TSA', 'TREFMNAV', 'TREFMXAV', 'BTRANMN', 'Vcmx25Z', 'FSH', 'VEGWP', 'FCTR', 'FCEV', 'FGEV',
         'FIRE', 'FSR', 'FIRA', 'FSA', 'GSSUNLN', 'GSSHALN', 'TSKIN', 'GPP', 'NPP', 'AGNPP', 'TOTVEGC',
         'NPP_NUPTAKE', 'AR', 'HR', 'HTOP', 'GRAINC_TO_FOOD', 'NFERTILIZATION'
 hist_fincl3 = 'FSR', 'H2OSNO', 'Q2M', 'SNOWDP', 'TSA', 'TREFMNAV', 'TREFMXAV', 'TG', 'QRUNOFF',
         'FSH', 'FIRE', 'FIRA', 'FGR', 'EFLX_LH_TOT', 'RH2M', 'TLAI', 'SOILWATER_10CM', 'TOTSOILLIQ',
         'TOTSOILICE', 'U10', 'TSOI_10CM', 'QIRRIG_DRIP', 'URBAN_HEAT', 'WASTEHEAT', 'TSKIN', 'GPP', 'NPP',
         'AR', 'HR', 'DWT_CONV_CFLUX_PATCH', 'WOOD_HARVESTC', 'DWT_WOOD_PRODUCTC_GAIN_PATCH', 'SLASH_HARVESTC', 'COL_FIRE_CLOSS', 'DWT_SLASH_CFLUX_PATCH', 'FROOTC:I',
         'HTOP', 'GRAINC_TO_FOOD'
 hist_fincl4 = 'PCT_GLC_MEC', 'QICE_FORC', 'TSRF_FORC', 'TOPO_FORC', 'PCT_NAT_PFT', 'PCT_LANDUNIT', 'FSNO_ICE',
         'C14_SOILC_vr', 'SOILC_vr', 'SOILN_vr', 'CWD_C_vr', 'LIT_MET_C_vr', 'LIT_CEL_C_vr', 'LIT_LIG_C_vr',
         'LIT_MET_N_vr', 'LIT_CEL_N_vr', 'LIT_LIG_N_vr', 'CWD_N_vr', 'TOTLITC:I', 'TOT_WOODPRODC:I', 'TOTSOMC:I',
         'TOTVEGC:I', 'PCT_CFT', 'CROPPROD1C:I'
 hist_fincl5 = 'TOTSOMC:I', 'TOTSOMC_1m:I', 'TOTECOSYSC:I', 'TOTVEGC:I', 'WOODC:I', 'TOTLITC:I', 'LIVECROOTC:I',
         'DEADCROOTC:I', 'FROOTC:I'
 hist_mfilt(1) = 1
 hist_mfilt(2) = 1
 hist_mfilt(3) = 1
 hist_mfilt(4) = 1
 hist_mfilt(5) = 1
 hist_nhtfrq(1) = 0
 hist_nhtfrq(2) = 0
 hist_nhtfrq(3) = 0
 hist_nhtfrq(4) = -8760
 hist_nhtfrq(5) = -8760
 hist_type1d_pertape(1) = ' '
 hist_type1d_pertape(2) = ' '
 hist_type1d_pertape(3) = 'LAND'
 hist_type1d_pertape(4) = ' '
 hist_type1d_pertape(5) = 'LAND'

UPDATE: I had to remove crop and c14-related variables from the fincl statements for the above to work this time.

[slevis-lmwg]

#88 vs. #72 diagnostics IN PROGRESS on casper
#88 rerun last few decades with additional history output IN PROGRESS on derecho

[slevis-lmwg]

Land diagnostics posted at the top.
The rerun with additional history output should complete tonight.

[wwieder]

Here's a look at max LAI / PFT survival for this case: