Public IP Address orphaned when updating LoadBalancer type Service to a user-provided PIP from another RG

[jfchevrette]

What happened:

Updating an existing LoadBalancer ty kubernetes Service to bind it to a user-created PIP does not release the existing PIP when that PIP is in a different resource group

What you expected to happen:

The PIP which was previously bound to the service should be released and removed

How to reproduce it (as minimally and precisely as possible):

0. Set variables to use in future commands

export TEST_RG=test-orphaned-pip

1. Create a resource-group to test with

az group create \
  --location eastus \
  --resource-group $TEST_RG

2. Create an AKS cluster

az aks create \
  --resource-group $TEST_RG \
  --name my-cluster \
  --node-count 1 \
  --generate-ssh-keys

3. Export cluster credentials to a temporary file and set KUBECONFIG

az aks get-credentials \
  --resource-group $TEST_RG \
  --name my-cluster \
  -f /tmp/my-cluster.kubeconfig

export KUBECONFIG=/tmp/my-cluster.kubeconfig

4. Create a test loadbalancer service on the cluster

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: my-loadbalancer-service
spec:
  ports:
  - port: 80
  type: LoadBalancer
EOF

5. Observe the service's external IP address is created and assigned
6. Get the cluster's identity principal and note it

az aks show \
  --resource-group $TEST_RG \
  --name my-cluster \
  --query "identity.principalId" \
  --output tsv

7. Create a PIP in $TEST_RG

az network public-ip create \
  --resource-group $TEST_RG \
  --name my-loadbalancer-service

8. Get the PIP's ID

az network public-ip show \
  --resource-group $TEST_RG \
  --name my-loadbalancer-service \
  --query "id" \
  --output tsv

9. Assign the Network Contributor role to the cluster identity principal onto the PIP

az role assignment create \
  --assignee <PRINCIPAL_ID> \
  --role "Network Contributor" \
  --scope <PIP_ID>

10. Annotate the loadbalancer service so that it is reassigned to our user-created PIP

kubectl annotate svc my-loadbalancer-service \
  service.beta.kubernetes.io/azure-load-balancer-resource-group=$TEST_RG \
  service.beta.kubernetes.io/azure-pip-name=my-loadbalancer-service

11. Observe that the service has been reconciled and now has our PIP assigned
12. Observe that the original PIP remains untouched in the cluster's node resource group

Note: If the user-created PIP is created in the cluster's node resource group, the PIP gets released properly as expected. However according to documentation it is a bad practice to create resources in this RG

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

[jfchevrette]

I've been looking at the code and while this is not my area of expertise, I'm starting to think the issue lies in that the code looks at the Service object, sees the annotation service.beta.kubernetes.io/azure-load-balancer-resource-group and operates on the Resource Group specified, disregarding the possibility that prior to the Service object update the annotation may not have been there and thus it never looks at the PIPs in the cluster's Resource Group

The breadcrumbs I followed are as follow:

• Add the annotations to the Service object
• UpdateLoadBalancer is called
• which in turn calls reconcileService
  • reconcileService does a bunch of things and eventually calls reconcilePublicIPs
  • reconcilePublicIPs finds out what Resource Group to operate on using by calling getPublicIPAddressResourceGroup
    • getPublicIPAddressResourceGroup checks if service.beta.kubernetes.io/azure-load-balancer-resource-group annotation is set and uses its value as the RG
  • reconcilePublicIPs lists PIPs in that resource group and iterates on them
    • it calls reconcilePublicIP which is ultimately responsible for deleting PIPs (among other things)

All along, it appears that the service object (copy) that is passed through is the most recent one with the annotations. If that is indeed the case, the code should probably somehow get a copy of the old service object and consider that when determining what Resource Group to operate on (in getPublicIPAddressResourceGroup?)

[jfchevrette]

I also want to reiterate: If the Service is deleted after it has been annotated and reconciled with the user-provided PIP from the other RG, the original PIP remains and is never deleted