[Feature] Kuberay RayJob MultiKueue adapter

[mszadkow]

What type of PR is this?

/kind feature

What this PR does / why we need it:

We want to be bale to run Kuberay workloads RayJob in MultiKueue.
RayCluster will come as separate PR.

Which issue(s) this PR fixes:

Relates to #3822

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[mszadkow]

/ok-to-test

[netlify]

✅ Deploy Preview for kubernetes-sigs-kueue ready!

Name	Link
🔨 Latest commit	106712f
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/677ff77b2045fe0008f0ab24
😎 Deploy Preview	https://deploy-preview-3892--kubernetes-sigs-kueue.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[mszadkow]

So far it's the version without managedBy.
We don't have a ray-operator image that supports managedBy, that would have to be a custom one...
latest -> 1.2.2, which is the last release

[mszadkow]

this one is for us people working on macOS, it's vital for development, not so much for prod - I think it should stay

[mszadkow]

/retest

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mszadkow
Once this PR has been reviewed and has the lgtm label, please assign tenzen-y for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mszadkow]

Although the PR for Kuberay RayJob was ready for review before New Year, I couldn’t finish to work on it due to peculiar issue.

Only CI integration tests failed (problem not spotted locally), but before running any test.

IT failed on “Running Suite: Multikueue” and stuck forever.

Further investigation and enabling ginkgo output gave little more details.


Namely Envtest failed on setup and with couple of different yet relating errors:


 unable to install CRDs onto control plane: unable to create CRD instances: unable to create CRD "pytorchjobs.kubeflow.org": Post "https://127.0.0.1:44817/apis/apiextensions.k8s.io/v1/customresourcedefinitions": http2: client connection lost
 
unable to install CRDs onto control plane: unable to create CRD instances: unable to create CRD "rayjobs.ray.io": Post "https://127.0.0.1:33909/apis/apiextensions.k8s.io/v1/customresourcedefinitions": http2: client connection lost

 unable to install CRDs onto control plane: unable to create CRD instances: unable to create CRD "rayjobs.ray.io": Post "https://127.0.0.1:37813/apis/apiextensions.k8s.io/v1/customresourcedefinitions": http2: client connection lost
 
unable to install CRDs onto control plane: unable to create CRD instances: unable to create CRD "workloads.kueue.x-k8s.io": Post "https://127.0.0.1:32965/apis/apiextensions.k8s.io/v1/customresourcedefinitions": http2: client connection lost

Those errors were distributed over all cluster routines (as clusters run in separate go routines).



During the investigation with Mykhailo we were able to boil down the issue to simple addition of “ray-operator” CRDS to the suite.
#3920 - was our testing ground


We also checked:


• extended CRD installation times for testEnv

• extended WorkerLostTimeout option in SetupControllers

• increase INTEGRATION_API_LOG_LEVEL verbosity - this gave some insight but basically we saw that ETCD lost connection without learning the cause:
  E0103 11:50:24.628853 28403 finisher.go:175] "Unhandled Error" err="FinishRequest: post-timeout activity - time-elapsed: 1.969052ms, panicked: false, err: context canceled, panic-reason: <nil>" logger="UnhandledError"


Finally what worked was to reduce the INTEGRATION_NPROCS.

Which is still not understood why, but we assume that maybe apiserver of envtest got overwhelmed and possibly test-infra requires resource update.

We don’t have any proof that this will work, but some people reportedly so similar issues (not the exact ones):
kubernetes-sigs/controller-runtime#1246 (comment)


Maybe any of you guys encounter similar issue in the past?

/cc @mimowo @PBundyra @tenzen-y

[mimowo]

Given that you had to reduce INTEGRATION_NPROCS I suspect the cluster running the integration tests is resource constrained. However, I'm wondering if INTEGRATION_API_LOG_LEVEL=1 or --output-interceptor-mode=none could be making a difference here?

[mimowo]

To check if the tests are stable:
/test pull-kueue-test-integration-main
/test pull-kueue-test-multikueue-e2e-main

[mszadkow]

Given that you had to reduce INTEGRATION_NPROCS I suspect the cluster running the integration tests is resource constrained. However, I'm wondering if INTEGRATION_API_LOG_LEVEL=1 or --output-interceptor-mode=none could be making a difference here?

Those were added later to observe what is happening (after the behaviour started), but good point I will remove them.

[mimowo]

/test pull-kueue-test-integration-main
/test pull-kueue-test-multikueue-e2e-main

[mimowo]

Can you try to revert it and post a link to the failure log?

[mimowo]

Actually, could you maybe split the PR into smaller parts:

1. integration tests (which could be helpful to prove this fix works as expected: [WIP] Update Jobeframework's IsActive() for RayJob #3949 (IIUC these changes in test/integration are not MultiKueue-specific anyway)
2. actual adapter + e2e tests (this PR)

[mszadkow]

Actually, could you maybe split the PR into smaller parts:

1. integration tests (which could be helpful to prove this fix works as expected: [WIP] Update Jobeframework's IsActive() for RayJob #3949 (IIUC these changes in test/integration are not MultiKueue-specific anyway)
2. actual adapter + e2e tests (this PR)

I guess it's no longer valid, as there is no fix, rather extension for MK