Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Daemonset to istio-cni and fixed the script #2782

Merged
merged 2 commits into from
Jul 4, 2024

Conversation

biswajit-9776
Copy link
Contributor

Pull Request Template for Kubeflow manifests Issues

  • Please include a summary of changes and the related issue.
  • List any dependencies that are required for this change.
  • Please delete the options that are not relevant.
  • The following checklist will help you to satisfy the requirements.

✏️ A brief description of the changes

I added Daemonset back to istio-cni-1-22/istio-install/base/install.yaml and fixed the hack script

📦 List any dependencies that are required for this change

My PR depends on #

🐛 If this PR is related to an issue, please put the link of the issue here.

The following issues are related, because ...

✅ Unit Test Checklist

  • 🛠️ Make sure you have installed kustomize == 5.2.1+
  • ✍️ Have you written new tests for your core changes, as applicable?
  • 🔄 Have you successfully run existing tests with your changes ?
  • 🚀 Have you successfully run existing and new tests with your changes ?

✅ Contributor checklist


You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you need to drop --cluster-specific such that the hack script works without a running cluster. Why are you using the default profile in line 66? Isn't there a CNI profile?

Copy link
Contributor Author

@biswajit-9776 biswajit-9776 Jul 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[biswa@fedora bin]$ istioctl profile list
Istio configuration profiles:
    ambient
    default
    demo
    empty
    minimal
    openshift
    openshift-ambient
    preview
    remote
    stable

I don't see it explicitly here and it's default in istio-cni-1-17 README.md:
https://github.com/kubeflow/manifests/tree/v1.8-branch/common/istio-cni-1-17#:~:text=%24%20cd%20%24ISTIO_NEW%0A%24%20istioctl%20profile%20dump%20default%20%3E%20profile.yaml

Copy link
Contributor Author

@biswajit-9776 biswajit-9776 Jul 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also I found this:
https://istio.io/latest/docs/setup/additional-setup/config-profiles/#:~:text=The%20components%20marked,minimal
Do you think CNI is a component of ambient profile instead of default?

Copy link
Member

@juliusvonkohout juliusvonkohout Jul 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, maybe it changed recently. Then let's drop only the --cluster-specific and test on your cluster whether it really works.

Copy link
Contributor Author

@biswajit-9776 biswajit-9776 Jul 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have pushed the commit for the updated script and here are the cluster resources:

[biswa@fedora manifests]$ kubectl get daemonsets -n kube-system
NAME             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
istio-cni-node   3         3         3       3            3           kubernetes.io/os=linux   103s
kindnet          3         3         3       3            3           kubernetes.io/os=linux   2m29s
kube-proxy       3         3         3       3            3           kubernetes.io/os=linux   2m30s

I also see 3 healthy istio-cni pods in kube-system namespace in each cluster node:

[biswa@fedora manifests]$ kubectl get pods -n kube-system
NAME                                         READY   STATUS    RESTARTS   AGE
coredns-76f75df574-db5lj                     1/1     Running   0          3m57s
coredns-76f75df574-pdml9                     1/1     Running   0          3m57s
etcd-kind-control-plane                      1/1     Running   0          4m12s
istio-cni-node-28r44                         1/1     Running   0          3m25s
istio-cni-node-rpx22                         1/1     Running   0          3m25s
istio-cni-node-tlg2b                         1/1     Running   0          3m25s
kindnet-bmfbv                                1/1     Running   0          3m57s
kindnet-mk64l                                1/1     Running   0          3m51s
kindnet-pjz6s                                1/1     Running   0          3m52s
kube-apiserver-kind-control-plane            1/1     Running   0          4m12s
kube-controller-manager-kind-control-plane   1/1     Running   0          4m12s
kube-proxy-k5rcg                             1/1     Running   0          3m51s
kube-proxy-qkbk7                             1/1     Running   0          3m57s
kube-proxy-zjvpn                             1/1     Running   0          3m52s
kube-scheduler-kind-control-plane            1/1     Running   0          4m12s

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think we should maybe check if the ip tables are modified?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, please check a few initcontainers

@biswajit-9776
Copy link
Contributor Author

biswajit-9776 commented Jul 3, 2024

I tried deploying two containers httpbin and sleep from https://github.com/istio/istio/blob/master/samples in default namespace with label istio-injection=enabled with CNI enabled in same worker node

2024-07-03T17:53:30.883563Z	info	cni	============= Start iptables configuration for sleep-7656cf8794-d4qxt =============
2024-07-03T17:53:30.900930Z	info	cni	Istio iptables environment:
ENVOY_PORT=
INBOUND_CAPTURE_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_OUTBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_EXCLUDE_INTERFACES=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=
ISTIO_META_DNS_CAPTURE=
INVALID_DROP=
2024-07-03T17:53:30.900969Z	info	cni	Istio iptables variables:
PROXY_PORT=15001
PROXY_INBOUND_CAPTURE_PORT=15006
PROXY_TUNNEL_PORT=15008
PROXY_UID=1337
PROXY_GID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=*
INBOUND_PORTS_EXCLUDE=15020,15021,15090
OUTBOUND_OWNER_GROUPS_INCLUDE=*
OUTBOUND_OWNER_GROUPS_EXCLUDE=
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_INCLUDE=
OUTBOUND_PORTS_EXCLUDE=15020
KUBE_VIRT_INTERFACES=
ENABLE_INBOUND_IPV6=false
DUAL_STACK=false
DNS_CAPTURE=false
DROP_INVALID=false
CAPTURE_ALL_DNS=false
DNS_SERVERS=[],[]
NETWORK_NAMESPACE=/var/run/netns/cni-c284a504-3a1c-91c1-af66-1ae70f7c9c1c
CNI_MODE=true
EXCLUDE_INTERFACES=
2024-07-03T17:53:30.901038Z	info	cni	Running iptables restore with: iptables-restore and the following input:
* nat
-N ISTIO_INBOUND
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp --dport 15008 -j RETURN
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -p tcp --dport 15020 -j RETURN
-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -p tcp ! --dport 15008 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -p tcp ! --dport 15008 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
COMMIT
2024-07-03T17:53:30.901172Z	info	cni	Running command (without nss): iptables-restore --noflush
2024-07-03T17:53:30.907036Z	info	cni	Running iptables restore with: ip6tables-restore and the following input:
2024-07-03T17:53:30.907204Z	info	cni	Running command (without nss): ip6tables-restore --noflush
2024-07-03T17:53:30.908814Z	info	cni	Running command (without nss): iptables-save
2024-07-03T17:53:30.911727Z	info	cni	Command output: 
# Generated by iptables-save v1.8.9 (nf_tables) on Wed Jul  3 17:53:30 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:ISTIO_INBOUND - [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp -m tcp --dport 15008 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -p tcp -m tcp ! --dport 15008 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -p tcp -m tcp ! --dport 15008 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT
# Completed on Wed Jul  3 17:53:30 2024
2024-07-03T17:53:30.911770Z	info	cni	============= End iptables configuration for sleep-7656cf8794-d4qxt =============
2024-07-03T17:53:30.882127Z	info	cni	============= Start iptables configuration for httpbin-86b8ffc5ff-cwfzb =============
2024-07-03T17:53:30.900954Z	info	cni	Istio iptables environment:
ENVOY_PORT=
INBOUND_CAPTURE_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_OUTBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_EXCLUDE_INTERFACES=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=
ISTIO_META_DNS_CAPTURE=
INVALID_DROP=
2024-07-03T17:53:30.900986Z	info	cni	Istio iptables variables:
PROXY_PORT=15001
PROXY_INBOUND_CAPTURE_PORT=15006
PROXY_TUNNEL_PORT=15008
PROXY_UID=1337
PROXY_GID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=*
INBOUND_PORTS_EXCLUDE=15020,15021,15090
OUTBOUND_OWNER_GROUPS_INCLUDE=*
OUTBOUND_OWNER_GROUPS_EXCLUDE=
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_INCLUDE=
OUTBOUND_PORTS_EXCLUDE=15020
KUBE_VIRT_INTERFACES=
ENABLE_INBOUND_IPV6=false
DUAL_STACK=false
DNS_CAPTURE=false
DROP_INVALID=false
CAPTURE_ALL_DNS=false
DNS_SERVERS=[],[]
NETWORK_NAMESPACE=/var/run/netns/cni-00b63c4c-afc0-4a23-bb27-6c58c0617297
CNI_MODE=true
EXCLUDE_INTERFACES=
2024-07-03T17:53:30.901055Z	info	cni	Running iptables restore with: iptables-restore and the following input:
* nat
-N ISTIO_INBOUND
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp --dport 15008 -j RETURN
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -p tcp --dport 15020 -j RETURN
-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -p tcp ! --dport 15008 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -p tcp ! --dport 15008 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
COMMIT
2024-07-03T17:53:30.901173Z	info	cni	Running command (without nss): iptables-restore --noflush
2024-07-03T17:53:30.907042Z	info	cni	Running iptables restore with: ip6tables-restore and the following input:
2024-07-03T17:53:30.907204Z	info	cni	Running command (without nss): ip6tables-restore --noflush
2024-07-03T17:53:30.908814Z	info	cni	Running command (without nss): iptables-save
2024-07-03T17:53:30.911755Z	info	cni	Command output: 
# Generated by iptables-save v1.8.9 (nf_tables) on Wed Jul  3 17:53:30 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:ISTIO_INBOUND - [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp -m tcp --dport 15008 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -p tcp -m tcp ! --dport 15008 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -p tcp -m tcp ! --dport 15008 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT
# Completed on Wed Jul  3 17:53:30 2024
2024-07-03T17:53:30.911783Z	info	cni	============= End iptables configuration for httpbin-86b8ffc5ff-cwfzb =============

@biswajit-9776
Copy link
Contributor Author

biswajit-9776 commented Jul 3, 2024

These logs of istio-cni-node present in the same worker node point out that ip-tables are configured for the pods that are created in those namespaces and saved as logs in the istio-cni-node

@juliusvonkohout
Copy link
Member

/lgtm
/approve

Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: juliusvonkohout

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow bot merged commit 37e4352 into kubeflow:master Jul 4, 2024
17 checks passed
hansinikarunarathne pushed a commit to hansinikarunarathne/kubeflow-manifests that referenced this pull request Jul 22, 2024
* Added Daemonset to istio-cni and fixed the script

Signed-off-by: biswajit-9776 <[email protected]>

* Removed --cluster-specific from script in hack

Signed-off-by: biswajit-9776 <[email protected]>

---------

Signed-off-by: biswajit-9776 <[email protected]>
Signed-off-by: hansinikarunarathne <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants