checklist.tex

\section{Best Practices Checklist}

Our overall recommendations are summarized in the checklist  presented on the following page, which should facilitate avoiding common errors and adhering to good practices.

\begin{Checklists*}[p!]
\begin{checklist}{Quantifying Uncertainty and Sampling Quality in Molecular Simulation}
\begin{itemize}
\item
  \textbf{Plan your study carefully by starting with pre-simulation sanity checks.}
  There is no guarantee that any method, enhanced or otherwise, can sample the system of interest.
  See Sec.\ \ref{sec:sanity}
    \begin{itemize}
    \item Consult best-practices papers on simulation background and planning/setup.
      See: \url{https://github.com/MobleyLab/basic_simulation_training}
    \item Estimate whether system timescales are known experimentally and feasible computationally based on published literature.
      If timescales are too long for straight-ahead MD, investigate enhanced-sampling methods for systems of similar complexity.
      The same concept applies to MC, based on the number of MC trial moves instead of actual time.
    \item Read up on sampling assessment and uncertainty estimation, from this article or another source (e.g., Ref. \cite{Grossfield2009}).
      Understanding uncertainty will help in the \emph{planning} of a simulation (e.g., ensure collection of sufficient data).
    \item Consider multiple runs instead of a single simulation.
      Diverse starting structures enable a check on sampling for equilibrium ensembles, which should not depend on the starting structure.
      Multiple runs may be especially useful in assessing uncertainty for enhanced sampling methods.
    \item Check and validate your code/method via a simple benchmark system.
      See: \url{https://github.com/shirtsgroup/software-physical-validation}
    \end{itemize}
    \vspace{-0.325\baselineskip} %Line spacing after a sub-list is too large
    
\item
  \textbf{Do not ``cherry-pick'' data that provides hoped-for outcomes.}
  This practice is ethically questionable and, at a minimum, can significantly bias your conclusions.
  Use all of the available data unless there is an objective and compelling reason not to, e.g., the simulation setup was incorrect or a sampling metric indicated that the simulation was not equilibrated.
  When used, sampling metrics should be applied uniformly to \emph{all} simulations to further avoid bias.
    
\item
\textbf{Perform simple, semiquantitative checks which can rule out (but not ensure) sufficient sampling.} It is easier to diagnose insufficient sampling than to demonstrate good sampling.  See Sec.\ \ref{sec:quick}.
    \begin{itemize}
    \item Critically examine the time series of a number of observables, both those of interest \emph{and others}.
      Is each time series fluctuating about an average value or drifting overall?
      What states are expected and what are seen?
      Are there a significant number of transitions between states?
    \item If multiple runs have been performed, compare results (e.g., time series, distributions, etc.) from different simulations.
    \item An individual trajectory can be divided into two parts and analyzed as if two simulations had been run.
    \end{itemize}
    \vspace{-0.325\baselineskip} %Line spacing after a sub-list is too large

\item
  \textbf{Remove an ``equilibration'' (a.k.a. ``burn-in'', or transient) portion of a single MD or MC trajectory} and perform analyses only on the remaining ``production'' portion of trajectory.  An initial configuration is unlikely to be representative of the desired ensemble and the system must be allowed to relax so that low probability states are not overrepresented in collected data.  See Sec.\ \ref{sec:equil}.
 
\item
  \textbf{Consider computing a quantitative measure of global sampling}, i.e., attempt to estimate the number of statistically independent samples in a trajectory.  Sequential configurations are highly correlated because one configuration is generated from the preceding one, and estimating the degree of correlation is essential to understanding overall simulation quality.
  See Secs.\ \ref{sec:global} and \ref{sec:autocorrelation}.

\item
  \textbf{Quantify uncertainty in specific observables of interest using confidence intervals.}
  The statistical uncertainty in, e.g., the \hyperref[def:arith_mean]{\emph{arithmetic mean}} of an observable decreases as more independent samples are obtained and can be much smaller than the \hyperref[def:exp_st_dev]{experimental standard deviation} of that observable.
  See Sec.\ \ref{sec:specific}.
 
\item
  \textbf{Use special care when designing uncertainty analyses for simulations with enhanced sampling methods.}
  The use of multiple, potentially correlated trajectories within a single enhanced-sampling simulation can invalidate the assumptions underpinning traditional analyses of uncertainty.
  See Sec.\ \ref{sec:enhanced}.

\item
\textbf{Report a complete description of your uncertainty quantification procedure, detailed enough to permit reproduction of reported findings.}
Describe the meaning and basis of uncertainties given in figures or tables in the captions for those items, e.g., "Error bars represent 95\% confidence intervals based on bootstrapping results from the independent simulations."
Provide expanded discussion of or references for the uncertainty analysis if the method is non-trivial.
We strongly urge publication of unprocessed simulation data (measurements/observations) and post-processing scripts, perhaps using public data or software repositories, so that readers can exactly reproduce the processed results and uncertainty estimates. 
The non-uniformity of uncertainty quantification procedures in the modern literature underscores the value of clarity and transparency going forward.

\end{itemize}
\end{checklist}
\end{Checklists*}