As an infra / recipe developer, I want to know which data processing tool we will use

[trey-stafford]

Options:

• Lugi (currently used by QGreenland)
• parsl/ray.io (currently used by ADC/PDG)
• Apache beam (currently used by pangeo-forge)
• Prefect (previously used by pangeo-forge)

Evaluation criteria:

• Compatibility: can it be deployed on ADC k8s
• Flexibility/portability: can it be deployed on other systems (e.g., HPC, Google compute, etc.)
• Streaming and/or batch processing capabilities?

Evaluation spreadsheet: https://docs.google.com/spreadsheets/d/1f8CgOu1wXnaoAfEODrCkxNfY-Xq_as1AJPrPVK74lM0/edit#gid=0

• Create an example "small" workflow (seal tags, arctic circle)
• Create an example workflow that processes data and then runs the PDG workflow (viz-staging) ogdc-runner#38
• Create an example "large" workflow (drone imagery; @mbjones can you link to the drone imagery dataset we discussed in last month's meeting?)

[trey-stafford]

ADC docs related to using parsl for the PDG's vis-workflow

[trey-stafford]

Repo exploring the use of ray: https://github.com/QGreenland-Net/ray-exploration

Initial thoughts:

• Documentation is robust
• Multiple pathways for deployment / orchestration on kubernetes
• Feels "enterprise-y". Focus is on AI/ML, although Ray core can be used for a variety of more generic use-cases.

[trey-stafford]

Repo exploring the use of parsl: https://github.com/QGreenland-Net/parsl-exploration

Initial thoughts:

• Some elements don't feel "pythonic". E.g., "magic" kwargs on decorated functions (stdout and stderr on functions decorated wtih @bash_app)
• Limited documentation on how to use parsl on kubernetes
  • Workers require ability to connect to the machine running the parsl script, which means that machine must have a routable address. We work around this by sending the parsl script to kubernetes in a ConfigMap and running it with a Job. Is there a better way?
• Some error states result in 'orphaned' pods that go into a restart loop and need to be manually cleaned-up.
• Some issues around logging from e.g, @bash_app decorated functions. These write logs to files that have to be manually managed/inspected. There are some higher-level logging tools for monitoring parsl, however (see here)

[mfisher87]

We met about this today, but put the decision on hold while we continue to investigate ADC use cases on our workflow tech options.

https://docs.google.com/document/d/1ZcTcmVtUSjyWAy0TWnTWEWQ4Y2gqhub0yNaqVwpKxDY/edit

[trey-stafford]

We're planning on moving forward with Argo for OGDC workflows

[trey-stafford]

Confirmed that we plan to move forward with using Argo!