Report errors to BOTH ESMF Pet files AND log files

[ekluzek]

When things go wrong (and in an experimental model they always will) good and graceful error messaging is the most helpful thing to figure out the problem and get through it quickly. When running the model, too much logging can be problematic, but when debugging an error -- there can almost never be too much messaging on the error.

I realized that most of the problem I had with #524 was that the error messages only went to the ESMF PET files. In some of my cases I didn't see the PET files with the error. In others when you write by changing CREATE_ESMF_PET_FILES to TRUE it writes out files for each task and in some cases the error only shows up in one of them, so it's hard to find it amongst the files. And since the files have to be opened when the error occurs, if anything goes wrong with that process -- it's possible the error won't be reported anywhere. Since that XML variable above is normally FALSE you also don't see PET files for a successful run -- so you don't get used to seeing them and working with them.

As such it would be helpful to not only report the error to ESMF -- but also to the log files. I think this is a good usability enhancement that would be useful especially for new users. New users learn quickly to look at the log files and especially the cesm.log file. When error messaging doesn't show up there -- it's more difficult to track down.

So my suggestion is that instead of having errors done something like this:

                 if (ierr < 0) then
                    rc = ESMF_FAILURE
                    call ESMF_LogWrite(trim(subname)//' ERROR rpointer file open returns error', &
                         ESMF_LOGMSG_ERROR, line=__LINE__, file=__FILE__)
                    return
                 end if

It include a write to the log file as so

                 if (ierr < 0) then
                    rc = ESMF_FAILURE
+                  write(logunit,*) " error opening rpointer file "
                    call ESMF_LogWrite(trim(subname)//' ERROR rpointer file open returns error', &
                         ESMF_LOGMSG_ERROR, line=__LINE__, file=__FILE__)
                    return
                 end if

And since this is something to do lots of places in the code, even better would be to make a little subroutine or function that changes above to something like this (I suggest it would be added to mediator/med_utils_mod.F90 and would be similar to med_utils_ChkErr, except it would be a general error shutdown for any type of error rather than just an MPI error):

                 if (ierr < 0) then
                    call med_utils_ErrAbort( ErrMsg="rpointer file open returns error", line=__LINE__, file=__FILE__)
                    return
                 end if

I hope others will agree with me on this, but I'd also like to hear divergent thoughts. And completely open to suggestions on names and where this should go and other such details.