WIP: nd280 updates for 2024/2025

[NickHastings]

This branch has two minor feature updates for ND280 processing and some code cleanups.

[NickHastings]

Had a quick look but I don't see what needs to be done to get the failing checks to pass. Any hints?

[egede]

@NickHastings You are seeing a weakness of our CI. If the PR branch of your fork doesn't allow for the Ganga developers to edit (a box that you can tick or not), then the PEP8 compliance testing will fail. As all your changes are within the nd280 part, I think it is really up to you if you want to fix the PEP8 compliance or not. I suggest that we just merge this PR without it.

[egede]

Ganga has a general support for running jobs inside a container. It will be more flexible than what you implement here. I copy the documentation of the Apptainer class here for reference.

class Apptainer(GPIProxyObject)
 |  Apptainer(*args, **kwds)
 |  
 |      The Apptainer class can be used for either Apptainer or Docker images.
 |      It requires that apptainer is installed on the worker node.
 |  
 |      For Apptainer images you provide the image name and tag from Apptainer
 |      hub like
 |  
 |        j=Job()
 |        j.application=Executable(exe=File('my/full/path/to/executable'))
 |        j.virtualization = Apptainer("shub://image:tag")
 |  
 |      Notice how the executable is given as a `File` object. This ensures that it
 |      is copied to the working directory and thus will be accessible inside the
 |      container.
 |  
 |      The container can also be provided as a Docker image from a repository. The
 |      default repository is Docker hub.
 |  
 |        j.virtualization = Apptainer("docker://gitlab-registry.cern.ch/lhcb-core/lbdocker/centos7-build:v3")
 |  
 |        j.virtualization = Apptainer("docker://fedora:latest")
 |  
 |      Another option is to provide a `GangaFile` Object which points to a
 |      apptainer file. In that case the apptainer image file will be copied to
 |      the worker node. The first example is with an image located on some shared
 |      disk. This will be effective for running on a local backend or a batch
 |      system with a shared disk system.
 |  
 |        imagefile = SharedFile('myimage.sif', locations=['/my/full/path/myimage.sif'])
 |        j.virtualization = Apptainer(image= imagefile)
 |  
 |      while a second example is with an image located in the Dirac Storage
 |      Element. This will be effective when using the Dirac backend.
 |  
 |        imagefile = DiracFile('myimage.sif', lfn=['/some/lfn/path'])
 |        j.virtualization = Apptainer(image= imagefile)
 |  
 |      If the image is a private image, the username and password of the deploy
 |      token can be given like the example below. Look inside Gitlab setting for
 |      how to set this up. The token will only need access to the images and
 |      nothing else.
 |  
 |        j.virtualization.tokenuser = 'gitlab+deploy-token-123'
 |        j.virtualization.tokenpassword = 'gftrh84dgel-245^ghHH'
 |  
 |      Directories can be mounted from the host to the container using key-value
 |      pairs to the mounts option. If the directory is not available on the host, a
 |      warning will be written to stderr of the job and no mount will be attempted.
 |  
 |        j.virtualization.mounts = {'/cvmfs':'/cvmfs'}
 |  
 |      By default the container is started in apptainer with the `--nohome`
 |      option. Extra options can be provided through the `options` attribute. See
 |      the Apptainer documentation for what is possible.
 |  
 |      If the apptainer binary is not available in the PATH on the remote node - or has a different name,
 |      it is possible to give the name of it like
 |  
 |        j.virtualization.binary='/cvmfs/oasis.opensciencegrid.org/mis/apptainer/current/bin/apptainer'
 |  

[NickHastings]

@egede thanks for the info about the Apptainer class. I knew there would be a cleaner way to do this but I needed to get something up and running on a very short timescale, and my little hack that you see above fitted in with that. I'll back that commit out of the PR shortly.

[egede]

If what you want is to run inside a centos7 container for a Grid job (or in fact anywhere with /cvmfs access), there is a very efficient way of doing this. For a given job,

j.virtualization = Apptainer()
j.virtualization.image = '/cvmfs/cernvm-prod.cern.ch/cvm4'

is all that is required. It takes advantage of that this cvmfs location has an unpacked container image. Read the documentation for some more information about how this works.