[Inputs] Adding spin-up and restart capabilities

[MaggieMarvin]

From our discussion today, having the capability to spin-up and restart canopy-app seems like it could be very beneficial and may help to address current limitations.

For example, a short spin-up period could manage assumptions in the leaf age calculation that affect the first timestep (specifically, that past LAI = current LAI everywhere).

Spin-up and restart could also be a way to incorporate historical temperature averages (24-hour or 240-hour, etc) into the biogenic emissions calculations as done in MEGAN, to replace the instantaneous temperatures that are used currently as placeholders in canopy-app.

[drnimbusrain]

@MaggieMarvin I really like this idea, and hopefully you can help work on this as it is really important for biogenics.

I don't think its critical immediately for @quaz115's leafage PR, and for the initial time step past LAI = current LAI condition, as we still always want to maintain the functionality to run canopy-app at a single time (and potentially at a single point), for which there would not be a previous LAI value. The past LAI = current LAI condition is still valid, as seen by: https://github.com/quaz115/canopy-app-forkqzr/blob/develop/src/canopy_utils_mod.F90#L357

I agree though the spin-up/restart capability would be more important for the ability for canopy-app to handle historical variables, like past LAI, leaf temperature, and PAR for biogenics.

I think the following could be tentative start to the plan, and we can gleam off of MEGAN2 in HEMCO:

1. Add capability of outputting restart files from canopy-app, so that it could be restarted and/or run for a time-period to spin-up historical leaf temperatures and PAR, and for previous LAI in warm start. Need to write out the LAI, and historical leaf temperature and PAR for step 2.
2. Add capability/option for canopy-app to be restarted from a warm restart files (i.e., search and read historical average variables) upon run sequence. See https://github.com/geoschem/HEMCO/blob/main/src/Extensions/hcox_megan_mod.F90#L17
3. The historical variables and previous LAI from restart file are then updated as short-term (24-hr) and long-term (multiday) rolling averages in canopy-app going forward in model time step, as in HEMCO: https://github.com/geoschem/HEMCO/blob/main/src/Extensions/hcox_megan_mod.F90#L1000
4. If cold start, and no restart is found, then model still initializes as past LAI = current LAI (valid) and the historical leaf temp and PAR are zero, such that it starts from instantaneous values.

@MaggieMarvin @quaz115 @zmoon I think this could be moved up in priority for biogenics as its a larger uncertainty in canopy-app compared to current MEGAN2 and MEGAN3, and would appreciate any help @MaggieMarvin and @quaz115 if you agree.

[drnimbusrain]

@MaggieMarvin Maybe you can help get this part started with the tentative plan above, with respect to your other tasks of course. I think this is likely a remaining big part of the canopy-app overall and the biogenics prediction. This will impact the your upcoming biogenic emission evaluations, coupling to box-model predictions, and papers/presentations on this. Of course I could help you get started.

[MaggieMarvin]

I am setting up monthly simulations with canopy-app and have been able to cut down the runtime substantially for one day by cropping the domain, reducing the number of vertical levels etc, but there's still a massive memory barrier that will increase with simulation length and can't really be avoided without introducing this restart capability, so I will go ahead and start working on it now.