[Feature] Support get k8s pod log for interactive RayJob

[Moonquakes]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Currently, RayJob supports interactive submission, but there is a problem that without the submitter pod, the log information of RayJob cannot be obtained on the k8s cluster. Is there any solution?

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[rueian]

Hi @Moonquakes,

You can find job logs at /tmp/ray/session_date/logs/job-driver-{job_id}.log in your ray cluster pods.

[Moonquakes]

Hi @rueian, Thanks for your reply, but when RayJob is killed, the head node no longer exists, and in normal mode I can always get the corresponding log information through the submitter pod. Is there any way to achieve a similar effect in interactive mode?

[rueian]

Hi @Moonquakes,

The job driver logs are streamed to the stdout of the submitter (the ray job submit cli tool) and are stored in the logs folder I mentioned earlier. These are the only two places the logs present.

If you don’t have the former logs stored somewhere, then persisting the latter somehow is the only choice available right now. I would suggest you follow the tutorial here for persisting them https://docs.ray.io/en/master/cluster/kubernetes/user-guides/persist-kuberay-custom-resource-logs.html