Make it possible to pass in labels and annotations that the operator will apply to all dynamically created objects (via the Go templates).

[spkane]

Describe the enhancement you'd like to see

We have a use case where we need the Testkube operators to create all objects in the cluster (test jobs, etc) with a specific annotation. In our case, this is a special ArgoCD annotation that tells ArgoCD to ignore these resources, which it is unaware of.

At the moment, there is no way to modify the default templates that are used by Testkube operator to create these objects without completely overwriting the Go templates that Helm installs.

This means we either must overwrite the template completely and then actively track upstream changes that might be introduced or add the job template modifications to every test we write. We want a way via the Helm values file to tell the operator to apply a set of custom labels and annotations to each object that it creates dynamically.

Additional context

In our scenario, Testkube was installed using helm chart and kustomize through ArgoCD. Because of the argoCD setting we use, ArgoCD attempts to prune every Testkube test job that gets dynamically created by the operator. This leads to a race condition that results in about 25% of the test pods being removed before Testkube can gather the test logs. To work around this, we added a job-template patch to each test, but we would prefer a way to fix this at a lower level.

We could take other approaches to work around this issue, but this functionality seems helpful in various use cases beyond just this one.

cc/ @manidharanupoju24

[spkane]

#694 provides one approach to doing this via the Helm chart alone (for the default job template). Although this works, I am not convinced it is ideal (but possibly good enough).

Off the top of my head, the cleanest approach is to have a way to add a list of labels and annotations to the operator config so that if they are defined, the operator will add those to any object that it creates.

That being said, this PR works by only changing the chart and not touching the Testkube code.

[vsukhin]

@spkane technically, operator can use cronjob template filed from Test CRD and adjust created con jobs. What other resouces do you need to modify?

[spkane]

@spkane technically, operator can use cronjob template filed from Test CRD and adjust created con jobs. What other resouces do you need to modify?

@vsukhin I am not sure exactly what you are suggesting here.

At the moment, we only need to modify the test jobs that are being created by the operator, but in general, I would say that we want to be able to tell the operator to apply a specific annotation to ALL objects that it creates in the cluster, and if one is going to support this, it makes sense to simply allow a list of annotations and labels to be defined and applied.

[vsukhin]

Right now, operator creates only cron jobs and you can adjust them using labels and annotations. We will suppport this for any new objects we will create in operator

[frederikb]

@spkane Can you share which of the two options you're using as a workaround for now? We're evaluating testkube in combination with ArgoCD and are running into the same issues:

• Maintain a complete custom job template while keeping it up to date with any upstream changes
• Overriding and adding the ArgoCD annotations on each and every test

@vsukhin
As a way forward and seeing that #694 could break existing installations and is therefore not likely get merged. How about the following?

Looking at the implementation there is a difference between how job template configurations are handled:

• the job template overridable via Helm must be a complete definition
• whereas the job template extension configured via Test CR is merged into the existing rendered YAML as a patch

How about reusing the patch mechanism and allow defining a job patch in the Helm chart as well?
Then the final job YAML would be: job template + job patch from Helm + job patch from Test CR

1. Existing installations are not impacted
2. The patch mechanism can be reused
3. We can define our installation wide overrides once (DRY) - e.g., to add required ArgoCD annotations

[vsukhin]

@frederikb Sounds like a good idea, I support it

[spkane]

@frederikb We ended up going with simply adding the ArgoCD annotation patch into each test since it was annoying, but much less work in the end than trying to fork the helm charts.

I like your idea for handling this, although it will probably require applying the patches in a well-known order, so the merge behavior is consistent, but it is certainly cleaner than #694, which shouldn't break anything, but is mostly a hack.

The core problem here is that the test Job template is a Go template used directly by the operator and not a helm template. Another approach to solving this problem might be to support passing in additional key/value pairs to the operator via the Helm manifest which it can then use with the go template, instead of trying to mix Helm templating with Go templating.

[vsukhin]

should be solved by @emil2k by moving template to helm templates

[frederikb]

Hi @vsukhin, any hint or recommendation on how to solve the issue of this task now that the templates have been migrated to Helm?

I see that jobPodAnnotations was recently added, however that (of course) does not apply to the top level annotations of the jobresources.

[vsukhin]

hey @frederikb you can add one more variable, if this is not enough. should be pretty similar change

[frederikb]

@vsukhin Sure, I've created a PR as suggested: #873