This is a prototype built to try out some ideas about managing and analyzing large ensembles of simulations. Here, n-body problems are used as a concrete test case. The traditional approach to this problem is to create a scirpt which build a tree of directories, one per run, with human-readable names, containing configuration files and queue system job scripts for each run, and then a script to submit the required jobs. A simulation code would be compiled to a binary, the jobs submitted to the queue system, and hopefully everything runs as intended. The loose coupling between the stages of the process means it is difficult to supervise the process and modify it. In many places, redundant copies of the same information are created that neeed to be kept in sync, for example; the directory names, configuration file parameters, and scripts that generate these. This approach also makes strong assumptions about the compute platform, being more or less a traditional HPC cluster.
In this system, currently on the second version, Singularity containers are used to provide a repeatable and portable environment for building and executing code. The idea is that this is particularly handy when dealing with myriad dependencies, allows better testing before deployment, and enables reproducibility. As opposed to traditional approaches to running simulations, one-per-run configuration files and directories are also done away with by moving all configuration details into code, and interfacing with the simulation code itself as a library wrapped in an object-oriented approach. Simulation runs are identified with a unique key generated by a hash function from their parameters. Once these are organized into a library object, metadata is primary accessed by importing a code library. Any simulation output files written are named based on the unique model hash key, making unique naming simple in a flat namespace which will eventually mean freedom from the dependence on block storage devices. Again in contrast to traditional approaches, the organization of output on the storage device is not intended to be human-readable. All interface to the data is done through code, which opens up much more flexibility.
-
Singularity Container The entire system is run from a Singularity container (like Docker, but more HPC cluster-friendly) built on the public Sigularity-Hub system. Everything that goes into this container is in this Git repo, or other open ones.
- Singularity https://sylabs.io
- Singularity-Hub collection for this repo: https://singularity-hub.org/collections/3350
-
SGE job script Allocates nodes and set the containers running on the cluster via the queue system.
-
Driver Class A class which, informed by the Campaign object, directs execution of the simulation in a Model object, triggering I/O, checkpointing, restarts, and wall clock limits from the queue system. If the queue system is used to allocate a job array, the wall clock limited driver runs a single model of a job. If a single block of nodes is allocated at once for MPI-parallel mode a MPI-enabled load-balancer master-slave system wraps the wall clock limited driver to advance all the models in the campaign across the allocated cores.
-
Campaign Class A class which implements a library of Model objects. Primarily provides indexes to look up Model objects by parameter values.
-
Model Class A class which hold a full set of parameters and initiates, restarts, integrates, and checkpoints a REBOUND/REBOUNDx simulation. Importantly, each Model object computes a unique MD5 hash from the complete set of parameters, which yield a globally unique identifier. Data is stored with the identifier as the key, which can be looked up by parameter values in the Campaign class.
-
REBOUND and REBOUNDx Model objects create and call methods from REBOUND n-body integrator objects. REBOUNDx, with some additional effects implemented (in crbx/) is used to provide additional forces beyond gravity.
- REBOUND https://rebound.readthedocs.io
- REBOUNDx https://reboundx.readthedocs.io
One problem implemented in this system is the long-term stability of resonant chains of planets, in the style of:
Matsumoto, Y., Nagasawa, M., & Ida, S. (2012). The orbital stability of planets trapped in the first-order mean-motion resonances. Icarus, 221(2), 624–631. http://doi.org/10.1016/j.icarus.2012.08.032
The first set of results generated confirms the basic pattern in Matsumoto et al. (2012), that although chains in mean motion resonance (MMR) have much longer stability times than non-resonant ones with similar spacing, there exists a critical length of chain beyond which the stability timescale rapidly decreases. For planets of planet mass / star mass ratio q=1e-5 they give critical chain lengths of 8 in 6:5 MMR and 4 in 7:6 MMR.
The data from these simulations performed with the old_v1 version on the orbit crossing time is:
where orange triangles indicate lower bounds.
Colin McNally www.colinmcnally.ca