Enable multi-architecture support, specifically for s390x, in CNAO and the associated CNIs.

[ashokpariya0]

Is your feature request related to a problem? Please describe:
Provide multi-platform support for CNAO and all CNIs supported by CNAO.

Describe the solution you'd like:
We would like to see multi-architecture support for CNAO and all of its supported CNIs, including:

multus
multusDynamicNetworks
linuxBridge
kubeMacPool
ovs
macvtap
kubeSecondaryDNS
kubevirtIpamController

Describe alternatives you've considered:

[ormergi]

@ashokpariya0 could you please elaborate on the motivation for supporting s390x arch?
Is there some tracking issue or enhancement proposal for supporting s390x?

[ashokpariya0]

We are enabling s390x (IBM Z system) support in all(most) kubevirt ecosystem components. Already kubevirt (virt-operator), supports s390x and now we are working on additional components like cnao, cdi(almost done), ssp, etc.

[oshoval]

Checked again, wrt auto bumper, i think in this specific case we dont need to update bumper scripts as this parameter is part of CNAO and not part of the components.
So your PR is fine.

What about all the other images, some of them arent multiarch right ?
We won't able to maintain it right now, but will prioritize it as a task for all the images once we can.
If you plan to contribute and able to give support if it breaks we might able to take your PR, thanks.
Maybe we first need to map all the required changes over all repos and just then start by fixing one by one.

Also please consider that kube-rbac-proxy need to be changed on https://github.com/k8snetworkplumbingwg/kubemacpool so if KMP is installed standalone or via CNAO it will be the same

WRT the proposed image, it seems safe to use it as https://github.com/brancz/kube-rbac-proxy is the one
https://github.com/openshift/kube-rbac-proxy is forked from

Wdyt about fixing it / opening an issue on https://github.com/openshift/kube-rbac-proxy please ?

@phoracek agree ? anything i missed ?
Thanks

[ashokpariya0]

Checked again, wrt auto bumper, i think in this specific case we dont need to update bumper scripts as this parameter is part of CNAO and not part of the components. So your PR is fine.

@oshoval Thank you for your feedback.

What about all the other images, some of them arent multiarch right ? We won't able to maintain it right now, but will prioritize it as a task for all the images once we can. If you plan to contribute and able to give support if it breaks we might able to take your PR, thanks. Maybe we first need to map all the required changes over all repos and just then start by fixing one by one.

Yes, you are right. We have noticed that most of the CNI images do not support multiple architectures, and we are in the process of identifying the necessary changes. Our initial focus is to get the CNAO operator running on the s390x architecture(for this we need to change kube-rbac-proxy image), and then we will address the other CNI images one by one, as you suggested. We are committed to actively contributing to this effort.

Also please consider that kube-rbac-proxy need to be changed on https://github.com/k8snetworkplumbingwg/kubemacpool so if KMP is installed standalone or via CNAO it will be the same

Okay, Noted.

WRT the proposed image, it seems safe to use it as https://github.com/brancz/kube-rbac-proxy is the one https://github.com/openshift/kube-rbac-proxy is forked from

Yes, That is correct.

Wdyt about fixing it / opening an issue on https://github.com/openshift/kube-rbac-proxy please ?

Sure, I will create the issue and follow up on it.

[oshoval]

Thank you

Yes, you are right. We have noticed that most of the CNI images do not support multiple architectures, and we are in the process of identifying the necessary changes. Our initial focus is to get the CNAO operator running on the s390x architecture(for this we need to change kube-rbac-proxy image), and then we will address the other CNI images one by one, as you suggested. We are committed to actively contributing to this effort.

Just to make sure, i meant other components that CNAO deploys, do we talk about the same?
(i believe we do)

kubeMacPool
kubeSecondaryDNS
kubevirtIpamController
linuxBridge
macvtap
multus
multusDynamicNetworks
ovs

[ashokpariya0]

Thank you

Yes, you are right. We have noticed that most of the CNI images do not support multiple architectures, and we are in the process of identifying the necessary changes. Our initial focus is to get the CNAO operator running on the s390x architecture(for this we need to change kube-rbac-proxy image), and then we will address the other CNI images one by one, as you suggested. We are committed to actively contributing to this effort.

Just to make sure, i meant other components that CNAO deploys, do we talk about the same?

kubeMacPool
kubeSecondaryDNS
kubevirtIpamController
linuxBridge
macvtap
multus
multusDynamicNetworks
ovs

Yes.
I tried all of the above CNIs, Below is my findings.
Multus : Able to deploy with latest image tag.
Multus Dynamic Networks Controller: We don't have mutli arch supported image for this CNI plugin. We need to build one.
linuxBridge- Image is multi arch supported but we see runtime issue(Marker binary is unavailable for s390x) with images.
macvtap: Image is multi arch supported but cp binray present inside image is not s390x compatible
ovs: We don't have mutli arch supported images for this CNI plugin.
KubeSecondaryDNS : We don't have multi arch image for this CNI plugin.

[oshoval]

Amazing thank you
what about kubeMacPool / kubevirtIpamController if possible as well please ?

[oshoval]

Please keep in mind for changes that are done, that we will also need ARM, so best if changes can be done in a robust manner (or at least strives to)
no need to test for ARM of course, just if some new builds are made, will be nice if the scripts can be robust please

[ashokpariya0]

Amazing thank you what about kubeMacPool / kubevirtIpamController if possible as well please ?

Regarding kubemacpool:- Image is multi arch supported However I see runtime issue with image as manager binary present inside kubemacpool-cert-manager init container is not compatible with s390x.
kubevirtIpamController: looks fine.

[oshoval]

Thanks a lot, we are here whatever needed

[oshoval]

Hi @zhlhahaha
Might interest you about ARM, as some of the changes might affect it as well

[oshoval]

Might worth please to consider a gist about how to emulate s390x so whoever need to sanity check / develop it can
https://www.qemu.org/docs/master/system/target-s390x.html

[ashokpariya0]

Might worth please to consider a gist about how to emulate s390x so whoever need to sanity check / develop it can https://www.qemu.org/docs/master/system/target-s390x.html

Sure, Thanks!

[zhlhahaha]

Hi @zhlhahaha Might interest you about ARM, as some of the changes might affect it as well

Let me take a look, thanks for reminding @oshoval

[ashokpariya0]

@oshoval Please check PR: #1917

[ashokpariya0]

@oshoval I created an issue on GitHub to enable multi-architecture image support for quay.io/openshift/origin-kube-rbac-proxy. You can find the issue here: Issue #113

[ashokpariya0]

@oshoval @RamLavi The last pushed upstream CNAO image (link) was 13 days ago. Since then, a couple of pull requests have been merged in the CNAO repository (specifically the last two). However, I haven't seen a new operator image published. Could you please help clarify the reason for this?

[oshoval]

Hi, we dont release on each PR, just once there is a need, if you need it we can release

[ashokpariya0]

@phoracek @oshoval @RamLavi
Please review the PR at #1923 when you get a chance.

[ashokpariya0]

@oshoval @phoracek could you please direct me to the GitHub repository where the image below is being prepared?
https://quay.io/repository/kubevirt/cni-default-plugins?tab=tags&tag=latest

[oshoval]

hack/components/bump-linux-bridge.sh
by this script i think, which runs on CNAO git actions (bump release action)

[ashokpariya0]

hack/components/bump-linux-bridge.sh by this script i think, which runs on CNAO git actions (bump release action)

Thanks, @oshoval , for the quick reply! I got my answer.

[ashokpariya0]

@oshoval registry-image https://github.com/kubevirt/cluster-network-addons-operator/blob/fca381a5d1d1b480ae21240fb36f3e311164aa9e/build/registry/Dockerfile#L1C6-L1C48
is built from https://github.com/openshift/operator-framework-olm/blob/master/operator-registry.Dockerfile ??

[oshoval]

you mean who build quay.io/openshift/origin-operator-registry:4.2.0 ?
i dont know by heart sorry, if i find i will update here

[oshoval]

Hi, about git actions not atm please, won't have time to work on that, lets first do the minimum first
about compilation, we can have benchmark and act accordingly please once we can compare, imho auto detection is good enough in any case (let me know please why not if you think otherwise)

[oshoval]

All your PRs that aren't merged yet, so will be easier to track,
if i miss anything / you would like to add new ones please let me know

Thanks a lot for your effort

• kubevirt/kubesecondarydns#75
• kubevirt/kubesecondarydns#73
• k8snetworkplumbingwg/kubemacpool#441
• kubevirt/cluster-network-addons-operator#1927
• kubevirt/cluster-network-addons-operator#1928
• kubevirt/bridge-marker#77
• k8snetworkplumbingwg/ovs-cni#336

EDIT
once we done, this is the list of what we should release / verify

• kubeMacPool
• kubeSecondaryDNS (released)
• kubevirtIpamController
• linuxBridge
• macvtap
• multus
• multusDynamicNetworks (release was created)
• OVS
• Bridge marker
• ipam-ext
• CNAO

[oshoval]

We miss macvtap in case you would like to address it please as well
Thanks

[ashokpariya0]

We miss macvtap in case you would like to address it please as well Thanks

Sure Noted, I will work on that.

[ashokpariya0]

@oshoval Once the PR #336 gets merged, I can submit the changes for s390x (which will be a small update). After that, we can request a new release for the OVS CNI.

[oshoval]

@oshoval Once the PR #336 gets merged, I can submit the changes for s390x (which will be a small update). After that, we can request a new release for the OVS CNI.

awesome thanks
k8snetworkplumbingwg/ovs-cni#336 is merged

[ashokpariya0]

@oshoval Once the PR #336 gets merged, I can submit the changes for s390x (which will be a small update). After that, we can request a new release for the OVS CNI.

awesome thanks k8snetworkplumbingwg/ovs-cni#336 is merged

I submitted s390x support PR for OVS CNI: k8snetworkplumbingwg/ovs-cni#337

[oshoval]

Thanks

Lets please also run locally docker / podman on the repos to make sure it does work for both for multi arch / cross etc
at least some of the combinations

if help is needed lets sync on slack, thanks

[ashokpariya0]

Thanks

Lets please also run locally docker / podman on the repos to make sure it does work for both for multi arch / cross etc at least some of the combinations

if help is needed lets sync on slack, thanks

Thanks @oshoval
I’ve considered that, and here’s what I do:

1. I build on amd64 and then test on s390x to validate cross-compilation.
2. I also build directly on s390x and test on the same architecture to ensure everything works natively.
3. Additionally, I test with both Docker and Podman to ensure compatibility across both container runtimes.

[oshoval]

Awesome, thanks a lot

[oshoval]

Hi, I didn't manage yet to have a system end to end that compiles and push everything in every combinations,
and seems i won't able to focus on that, so i can't maintain it myself atm.
It means that if problems related to multi arch will arise, we might need to ask your help,
will you able please to support it?
Our main effort will still be the stability of the main flow releases.
Thanks

[oshoval]

/cc @jschintag
since you helped in this effort, thanks

[ashokpariya0]

Hi, I didn't manage yet to have a system end to end that compiles and push everything in every combinations,
and seems i won't able to focus on that, so i can't maintain it myself atm.
It means that if problems related to multi arch will arise, we might need to ask your help,
will you able please to support it?
Our main effort will still be the stability of the main flow releases.
Thanks

@oshoval We're here to assist whenever you need extra hands with multi-arch or any related concerns. Just let us know, and we'll do our best to help!

[oshoval]

Thank you very much for all the effort and co-op

[jschintag]

No Problem, thank you @oshoval for your time and help in this.
And as @ashokpariya0 said if you have a multi-arch problem feel free to ping us.

[oshoval]

Can you please check that CNAO release jobs themselves support and will use "all" flag ?

[ashokpariya0]

Can you please check that CNAO release jobs themselves support and will use "all" flag ?

Sure, could you please direct me to the Prow jobs responsible for the release?

[oshoval]

please look on post submit of cnao under project infra or on git actions
if you dont manage i will try to help

[ashokpariya0]

Can you please check that CNAO release jobs themselves support and will use "all" flag ?

Done, PR link: #1956

[ashokpariya0]

All your PRs that aren't merged yet, so will be easier to track, if i miss anything / you would like to add new ones please let me know

Thanks a lot for your effort

• kubevirt/kubesecondarydns#75
• kubevirt/kubesecondarydns#73
• k8snetworkplumbingwg/kubemacpool#441
• kubevirt/cluster-network-addons-operator#1927
• kubevirt/cluster-network-addons-operator#1928
• kubevirt/bridge-marker#77
• k8snetworkplumbingwg/ovs-cni#336

EDIT once we done, this is the list of what we should release / verify

• kubeMacPool
• kubeSecondaryDNS (released)
• kubevirtIpamController
• linuxBridge
• macvtap
• multus
• multusDynamicNetworks (release was created)
• OVS
• Bridge marker
• ipam-ext
• CNAO

EDIT UPDATE:
We need to merge the following PRs to complete this effort::
kubevirt/macvtap-cni#130

[oshoval]

Hi, KMP was bumped on CNAO
Please check / tick the list above, once all are tested, lets create a new CNAO release
Thanks

[ashokpariya0]

Hi, KMP was bumped on CNAO Please check / tick the list above, once all are tested, lets create a new CNAO release Thanks

Yes, it's all done. I've verified all the CNI plugins on s390x, so we’re good to proceed with the new CNAO release.

A couple of points to note for our reference:

1. We’ve pinned the Multus Dynamic version to v0.3.4 with PR #1961, which does not support multi-architecture. The next version does support it, but we are sticking with v0.3.4 for now.
2. We don’t have multi-architecture support for passt-binding (but as we plan to remove it anyway, this isn’t a major concern ??).

Everything else looks good.

[RamLavi]

We’ve pinned the Multus Dynamic version to v0.3.4 with #1961, which does not support multi-architecture. The next version does support it, but we are sticking with v0.3.4 for now.

@maiqueb do you plan to issue a new release for multus-dynamic?

[oshoval]

Thanks Ashok
About [2] we are good, it will be removed, and returned in other repo or so, and there it might be good to support multi arch.
But this @nirdothan and @EdDev will decide.
About [1] I think we don't need to be blocked imho unless Miguel plans soon to release a new release (assuming the blocker was fixed)
EDIT - which will be the best of course

[ashokpariya0]

Thanks Ashok About [2] we are good, it will be removed, and returned in other repo or so, and there it might be good to support multi arch. But this @nirdothan and @EdDev will decide. About [1] I think we don't need to be blocked imho unless Miguel plans soon to release a new release (assuming the blocker was fixed)

Okay, Thanks for the information. We can make a decision based on Miguel's response.

[oshoval]

multus dynamic 0.3.6 was merged

[oshoval]

Hi Ashok, thanks for the awesome effort,
please close this ticket once you good with it.

[ashokpariya0]

Hi Ashok, thanks for the awesome effort, please close this ticket once you good with it.

Sure

[ashokpariya0]

Thank you everyone for all your help! A special thanks to @oshoval for your outstanding support, initiative, and attention to detail. I truly appreciate your efforts in ensuring these were merged on time. Closing this issue as it’s now complete.