add section on testing and answer files

doc/sundials_developers/source/development/Checklist.rst

@@ -81,3 +81,6 @@ system, etc. developers should adhere to the following checklist.
    only. Additionally, the ``SUNDIALS_DEBUG_PRINTVEC`` macro should be used to
    ``#ifdef`` out calls to the generic vector print functions ``N_VPrint`` and
    ``N_VPrintFile`` used for debugging purposes.
+
+#. If answer files changed, and it is expected/desired, then update the `.out` files
+   that are embedded in the `examples/` directory AND the :ref:`"answers" repository<>`. 

doc/sundials_developers/source/testing/Answers.rst

@@ -0,0 +1,98 @@
+..
+   -----------------------------------------------------------------------------
+   SUNDIALS Copyright Start
+   Copyright (c) 2002-2023, Lawrence Livermore National Security
+   and Southern Methodist University.
+   All rights reserved.
+
+   See the top-level LICENSE and NOTICE files for details.
+
+   SPDX-License-Identifier: BSD-3-Clause
+   SUNDIALS Copyright End
+   -----------------------------------------------------------------------------
+
+.. _Answers:
+
+Answer Files
+============
+
+When solving ODEs/DAEs and nonlinear algebraic equations numerically, there is not always a known
+analytical solution for the problem to compare with. Additionally, there may be many paths to a
+correct solution and some may be less efficient than others. This presents an issue for testing
+SUNDIALS. How do we ensure SUNDIALS is working correctly if we cannot write down what "correct" is?
+To solve this problem SUNDIALS has tests (which often also serve as example codes) that have been
+verified by the author to be functioning correctly. These tests produce output which is stored in a
+file known as the "answer" file (aka "out" or "output" file). The output typically consist of the
+solution and/or some quantities derived from the solution as well as statistics about how the
+integration and/or solve (e.g., number of time steps). When changes are made to SUNDIALS, we ensure
+that these output files do not change unless it is expected/desired. Practically, this is ensured
+by our :ref:`CI testing <CI>`.
+
+Due to difference in microarichtectures and the nature of floating point arithmetic, it is quite
+possible that the output generated on one machine may differ from the output generated on another.
+As such, we specify that the the answer files that are embedded in ``examples/`` (the ``.out``
+files) should match what is produced on the Jenkins CI machine.  We also have a `separate git
+repostiory <https://github.com/sundials-codes/answers>`_ which holds answers for other  machines,
+such as the GitHub Actions virtual machines. 
+
+
+Updating Answer Files
+---------------------
+
+Sometimes it is necessary to update the answer files so that the tests pass, if a change in output
+is expected/desired. Changing output files requires careful verification that the output is still
+"correct". The procedure for updating answer files is as follows:
+
+- For the ``.out`` files embedded in examples, simply overwrite the file with the new output. If the
+  tests still fail, then you will need to download the output file generated on the Jenkins CI
+  machine and commit it. Extracting this file from the Jenkins machine may require help from David
+  or Cody currently.
+
+- For the ``.out`` files in `sundials-codes/answers <https://github.com/sundials-codes/answers>`_:
+
+  #. Navigate to `test/answers` in the sundials repository. If there is nothing in this directory,
+     you need to run `git submodule init && git submodule update`.
+
+  #. Checkout the main branch: ``git checkout main``. 
+
+  #. Create a new branch off of main. It is ideal if you use a branch name that matches the name of
+     the branch that you are developing in sundials on: ``git checkout -b <branch name>``.
+
+  #. Update the relevant ``.out`` files in
+     ``test/answers/linux-ubuntu20.04-x86_64/gcc-9.4.0/<double|single|extended>`` (this is the files
+     used for the GitHub Actions CI). Like with the embedded `.out` files, you can try and use the
+     output generated on your machine, but it may be different enough from what is produced on the
+     GitHub actions to trigger a failure. You can download the output files generated on the GitHub
+     machines by clicking going to `https://github.com/LLNL/sundials/actions`_ finding your failing
+     test, clicking it, then at the bottom downloading the "artifacts" (you can also find your
+     failing test at the bottom of a PR). The downloaded zip file will be the SUNDIALS build
+     directory. The ``.out`` files you want are in
+     `build_that_you_downloaded_and_extracted/Testing/output`.
+
+  #. Create a pull-request in `sundials-codes/answers <https://github.com/sundials-codes/answers>`_
+     with your updates. 
+
+
+Adding New Answer Files
+-----------------------
+
+- For the ``.out`` files embedded in examples, simply overwrite the file with the new output. If the
+  tests running on Jenkins still fail, then you will need to download the output file generated on
+  the Jenkins CI machine and commit it. Extracting this file from the Jenkins machine may require
+  help from David or Cody currently.
+
+- For the ``.out`` files in `sundials-codes/answers <https://github.com/sundials-codes/answers>`_:
+
+  #. To add new answers, start by copying the relevant answers in ``linux-ubuntu20.04-x86_64`` to
+     your new directory (following the naming scheme outlined above).
+
+  #. Next Commit these files and open a PR to the "staging" branch.
+
+  #. Once the PR is merged, you can generate your new answer files and overwrite the ones you copied
+     from ``linux-ubuntu20.04-x86_64``.
+
+  #. Compare the diff and make sure it is reasonable, then commit.
+
+  #. Finally, open a new PR targeting the "staging" branch.
+
+  #. Eventually, "staging" will be merged into main.

doc/sundials_developers/source/testing/CI.rst

@@ -14,7 +14,7 @@
 GitHub CI Testing 
 =================
 
-There are two types of CI testing that we run on GitHub via `GitHub actions <https://github.com/LLNL/sundials/actions>`_:
+There are two categories of CI testing that we run on GitHub via `GitHub actions <https://github.com/LLNL/sundials/actions>`_:
 
 1. Comprehensive (excluding GPUs)
 2. Minimal/Short

doc/sundials_developers/source/testing/Overview.rst

@@ -0,0 +1,42 @@
+..
+   -----------------------------------------------------------------------------
+   SUNDIALS Copyright Start
+   Copyright (c) 2002-2023, Lawrence Livermore National Security
+   and Southern Methodist University.
+   All rights reserved.
+
+   See the top-level LICENSE and NOTICE files for details.
+
+   SPDX-License-Identifier: BSD-3-Clause SUNDIALS Copyright End
+   -----------------------------------------------------------------------------
+
+
+.. _Overview:
+
+Overview of SUNDIALS Testing
+============================
+
+We include several types of tests in SUNDIALS: unit tests, integration test and performance tests.
+These tests are run via as part of our :ref:`Continuous Integration suite <CI>`.
+
+Unit Tests
+----------
+
+The unit tests reside in two places: ``test/unit_tests`` and in files named ``test_`` within
+``examples/``. With few exceptions, SUNDIALS unit tests should return an exit code indicating if the
+test was successful or not. I.e., they should be self-contained and not require any output file to
+determine success or failure.
+
+Integration Tests
+-----------------
+
+The integration tests are dual purpose; they serve as tests and also examples for users. They are
+found in ``examples/``. The integration tests produce output that is checked against an output file,
+i.e. an answer file, to determine if the test passed or failed. See :ref:`Answers <Answers>` for details.
+
+
+Performance Tests
+-----------------
+
+These tests are benchmarks of SUNDIALS performance and are found in ``benchmarks/``. Refer to
+:ref:`Benchmarking <Benchmarking>` for more detail.