Cromwell

Introduction

Cromwell is a workflow engine which can be used to execute containers on a variety of platforms.

https://cromwell.readthedocs.io/

In particular, we are interested in using Cromwell to run ChRIS plugins using Singularity via a SLURM scheduler.

Thoughts

Actually, Cromwell is most similar in functionality as the pfcon + pman duo.

For our convenience, we have implemented a shim for pman to dispatch requests to Cromwell. In theory, this means all platforms supported by Cromwell are now also supported by ChRIS. But in reality, this of course takes some effort.

Architecture

pfcon, pman, and cromwell are running on a container host, where the cromwell container has access to a SLURM cluster via sbatch, squeue, and scancel commands. pfcon is responsible for localization and delocalization of data (i.e. moving data to and from a filesystem mounted by the SLURM cluster).`

Cromwell + SLURM

Here is an example configuration for Cromwell, for scheduling quick jobs with a default time of 5 minutes, 2 CPUs, 8GB mem, on the "short" partition, and the "mylab" account.

include required(classpath("application"))

backend {
  default = SLURM
  providers {
    SLURM {
      actor-factory = "cromwell.backend.impl.sfs.config.ConfigBackendLifecycleActorFactory"
      config {
        runtime-attributes = """
        Int runtime_minutes = 5
        Int cpus = 2
        Int requested_memory_mb_per_core = 4000
        String queue = "short"
        String account = "mylab"
        String docker
        String sharedir
        """

        submit-docker = """
        # https://cromwell.readthedocs.io/en/stable/tutorials/Containers/#job-schedulers
        # https://github.com/broadinstitute/cromwell/blob/develop/cromwell.example.backends/singularity.slurm.conf
        sbatch -J ${job_name} \
            -D ${cwd} -o ${out} -e ${err} -t ${runtime_minutes} \
            -p ${queue} -A ${account} \
            ${"-c " + cpus} \
            --mem-per-cpu ${requested_memory_mb_per_core} \
            chrispl_singularity_wrapper.sh \
                ${cwd}:${docker_cwd} \
                ${docker} ${job_shell} ${docker_script} ${sharedir}
        """

        kill = "scancel ${job_id}"
        check-alive = "squeue -j ${job_id}"
        job-id-regex = "Submitted batch job (\\d+).*"
      }
    }
  }
}

Singularity Wrapper Script

chrispl_singularity_wrapper.sh should be a wrapper script which executes ChRIS plugins using Apptainer. It can also have more features such as management of the Singularity image build cache. Here is a basic example satisfying the above usage:

#!/bin/bash -ex
cwd="$1"
image="$2"
shell="$3"
script="$4"
sharedir="$5"

export SINGULARITY_CACHEDIR=/work/singularity-cache
module load singularity/3.8.5
exec singularity exec --containall  -B "$cwd" -B "$sharedir:/share" "docker://$image" "$shell" "$script"

pman, pfcon, and Data Localization

pman should be configured with the environment variables:

SECRET_KEY=aaaaaaaa
CONTAINER_ENV=cromwell
CROMWELL_URL=http://example.com/
STORAGE_TYPE=nfs
STOREBASE=/some/path/on/nfs/server

/some/path/on/nfs/server should be a filesystem mounted in pfcon and in the SLURM cluster on the same path.