Skip to content
This repository has been archived by the owner on Mar 6, 2023. It is now read-only.

exposer routes rejected for a passthrough route with a secret to mount #165

Open
ntxt opened this issue Dec 13, 2021 · 1 comment
Open

exposer routes rejected for a passthrough route with a secret to mount #165

ntxt opened this issue Dec 13, 2021 · 1 comment
Labels
kind/bug

Comments

@ntxt
Copy link

ntxt commented Dec 13, 2021

What happened:
I have run the following script with a resource file below, the deployment and a passthrough route got created with the annotation pointing to a TLS secret. The secret gets created but the cert and key values are not populated, which makes the "docker-registry" fail looking for them (the secret is mounted to the container). The exposer route gets rejected with "HostAlreadyClaimed".
What you expected to happen:
The secret should get populated with the key and cert values, "docker-registry" should see and use them to secure the requests passed through by the route.
How to reproduce it (as minimally and precisely as possible):

ENV=staging #staging | live

oc new-project example-utils
oc apply -fhttps://raw.githubusercontent.com/tnozicka/openshift-acme/master/deploy/single-namespace/{role,serviceaccount,issuer-letsencrypt-${ENV},deployment}.yaml
oc create rolebinding openshift-acme --role=openshift-acme --serviceaccount="$( oc project -q ):openshift-acme" --dry-run -o yaml | oc apply -f -
sleep 5
oc apply -f docker-registry.yaml

# docker-registry.yaml
apiVersion: v1
kind: List
items:
- apiVersion: v1
  kind: Secret
  type: Opaque
  metadata:
    name: s3-docker-registry
  data:
    REGISTRY_STORAGE_S3_ACCESSKEY: ***
    REGISTRY_STORAGE_S3_SECRETKEY: ***
- apiVersion: v1
  kind: Service
  metadata:
    labels:
      app: docker-registry
      app.kubernetes.io/component: docker-registry
      app.kubernetes.io/instance: docker-registry
    name: docker-registry
  spec:
    ports:
    - name: 8443-tcp
      port: 443
      protocol: TCP
      targetPort: 8443
    selector:
      deployment: docker-registry
    sessionAffinity: None
    type: ClusterIP
  status:
    loadBalancer: {}
- apiVersion: v1
  kind: Service
  metadata:
    labels:
      app: docker-registry
      app.kubernetes.io/component: redis
      app.kubernetes.io/instance: redis
      app.kubernetes.io/name: redis
      app.kubernetes.io/part-of: docker-registry
      app.openshift.io/runtime-version: latest
    name: redis
  spec:
    ports:
    - name: 6379-tcp
      port: 6379
      protocol: TCP
      targetPort: 6379
    selector:
      app: redis
      deploymentconfig: redis
    sessionAffinity: None
    type: ClusterIP
  status:
    loadBalancer: {}
- apiVersion: route.openshift.io/v1
  kind: Route
  metadata:
    annotations:
      kubernetes.io/tls-acme: "true"
      acme.openshift.io/secret-name: "docker-registry-tls"
    labels:
      app: docker-registry
      app.kubernetes.io/component: docker-registry
      app.kubernetes.io/instance: docker-registry
    name: docker-registry
  spec:
    host: docker-registry.apps.example.io
    port:
      targetPort: 8443-tcp
    tls:
      insecureEdgeTerminationPolicy: Redirect
      termination: passthrough
    to:
      kind: Service
      name: docker-registry
      weight: 100
    wildcardPolicy: None
- apiVersion: apps/v1
  kind: Deployment
  metadata:
    generation: 15
    labels:
      app: docker-registry
      app.kubernetes.io/component: docker-registry
      app.kubernetes.io/instance: docker-registry
      app.kubernetes.io/part-of: docker-registry
    name: docker-registry
  spec:
    progressDeadlineSeconds: 600
    replicas: 1
    revisionHistoryLimit: 10
    selector:
      matchLabels:
        deployment: docker-registry
    strategy:
      rollingUpdate:
        maxSurge: 25%
        maxUnavailable: 25%
      type: RollingUpdate
    template:
      metadata:
        annotations:
          openshift.io/generated-by: OpenShiftNewApp
        creationTimestamp: null
        labels:
          deployment: docker-registry
      spec:
        containers:
        - env:
          - name: REGISTRY_HTTP_ADDR
            value: 0.0.0.0:8443
          - name: REGISTRY_HTTP_TLS_CERTIFICATE
            value: /certs/tls.crt
          - name: REGISTRY_HTTP_TLS_KEY
            value: /certs/tls.key
          - name: REGISTRY_STORAGE
            value: s3
          - name: REGISTRY_STORAGE_S3_ACCESSKEY
          - name: REGISTRY_STORAGE_S3_BUCKET
            value: example-docker-registry
          - name: REGISTRY_STORAGE_S3_REGION
            value: eu-west-1
          - name: REGISTRY_STORAGE_S3_SECRETKEY
          - name: REGISTRY_HTTP_SECRET
            value: a random secret generated by hand
          - name: REGISTRY_REDIS_ADDR
            value: redis:6379
          envFrom:
          - secretRef:
              name: s3-docker-registry
          image: registry
          imagePullPolicy: IfNotPresent
          name: docker-registry
          ports:
          - containerPort: 8443
            protocol: TCP
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /var/lib/registry
            name: docker-registry-volume-1
          - mountPath: /certs
            name: docker-registry-tls
            readOnly: true
        dnsPolicy: ClusterFirst
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext: {}
        terminationGracePeriodSeconds: 30
        volumes:
        - emptyDir: {}
          name: docker-registry-volume-1
        - name: docker-registry-tls
          secret:
            defaultMode: 420
            secretName: docker-registry-tls
- apiVersion: apps/v1
  kind: Deployment
  name: redis
  ...

openshift-acme logs:

I1212 23:38:51.169507       1 openshift-acme-controller.go:192] No kubeconfig specified, using InClusterConfig.
I1212 23:38:51.171856       1 openshift-acme-controller.go:236] Managing namespaces: []string{"example-utils"}
I1212 23:38:51.172286       1 openshift-acme-controller.go:272] Leaderelection ID is "openshift-acme-5cf885c959-j2gt5_a18cdf74-f8ea-4fc3-a7dc-6d4291dda970"
I1212 23:38:51.172337       1 leaderelection.go:242] attempting to acquire leader lease  example-utils/acme-controller-locks...
E1212 23:38:51.184525       1 leaderelection.go:331] error retrieving resource lock example-utils/acme-controller-locks: configmaps "acme-controller-locks" is forbidden: User "system:serviceaccount:example-utils:openshift-acme" cannot get resource "configmaps" in API group "" in the namespace "example-utils"
I1212 23:38:51.184547       1 leaderelection.go:247] failed to acquire lease example-utils/acme-controller-locks
I1212 23:39:07.385563       1 leaderelection.go:252] successfully acquired lease example-utils/acme-controller-locks
I1212 23:39:07.385628       1 openshift-acme-controller.go:329] Acquired leaderelection
I1212 23:39:07.385639       1 openshift-acme-controller.go:335] loglevel is set to "4"
I1212 23:39:07.385796       1 acme.go:89] Setting up kube informers for namespace "example-utils"
I1212 23:39:07.385903       1 route.go:136] Setting up route informers for namespace "example-utils"
I1212 23:39:07.385927       1 route.go:153] Setting up kube informers for namespace "example-utils"
I1212 23:39:07.385987       1 acme.go:114] Starting Account controller
I1212 23:39:07.385993       1 shared_informer.go:197] Waiting for caches to sync for account controller
I1212 23:39:07.386026       1 reflector.go:153] Starting reflector *v1.Secret (0s) from k8s.io/[email protected]/tools/cache/reflector.go:108
I1212 23:39:07.386030       1 reflector.go:153] Starting reflector *v1.ConfigMap (0s) from k8s.io/[email protected]/tools/cache/reflector.go:108
I1212 23:39:07.386040       1 reflector.go:188] Listing and watching *v1.Secret from k8s.io/[email protected]/tools/cache/reflector.go:108
I1212 23:39:07.386045       1 reflector.go:188] Listing and watching *v1.ConfigMap from k8s.io/[email protected]/tools/cache/reflector.go:108
I1212 23:39:07.386035       1 reflector.go:153] Starting reflector *v1.LimitRange (0s) from k8s.io/[email protected]/tools/cache/reflector.go:108
I1212 23:39:07.386077       1 reflector.go:188] Listing and watching *v1.LimitRange from k8s.io/[email protected]/tools/cache/reflector.go:108
I1212 23:39:07.386080       1 reflector.go:153] Starting reflector *v1.Service (0s) from k8s.io/[email protected]/tools/cache/reflector.go:108
I1212 23:39:07.386087       1 reflector.go:188] Listing and watching *v1.Service from k8s.io/[email protected]/tools/cache/reflector.go:108
I1212 23:39:07.386033       1 reflector.go:153] Starting reflector *v1.ReplicaSet (0s) from k8s.io/[email protected]/tools/cache/reflector.go:108
I1212 23:39:07.386211       1 reflector.go:188] Listing and watching *v1.ReplicaSet from k8s.io/[email protected]/tools/cache/reflector.go:108
I1212 23:39:07.386220       1 route.go:1347] Starting Route controller
I1212 23:39:07.386223       1 reflector.go:153] Starting reflector *v1.Route (0s) from k8s.io/[email protected]/tools/cache/reflector.go:108
I1212 23:39:07.386228       1 shared_informer.go:197] Waiting for caches to sync for route controller
I1212 23:39:07.386232       1 reflector.go:188] Listing and watching *v1.Route from k8s.io/[email protected]/tools/cache/reflector.go:108
I1212 23:39:07.390220       1 acme.go:180] Adding ConfigMap example-utils/letsencrypt-staging UID=edcf1aa5-e5e5-4d8a-a1af-b8be8eb23acb RV=46127840
I1212 23:39:07.395158       1 route.go:214] Adding Route example-utils/docker-registry RV=46128006 UID=9bc42f0c-30a2-4632-b587-bc890b1db48a
I1212 23:39:07.486130       1 shared_informer.go:227] caches populated
I1212 23:39:07.486159       1 shared_informer.go:204] Caches are synced for account controller 
I1212 23:39:07.486222       1 acme.go:271] Started syncing Account "example-utils/letsencrypt-staging"
I1212 23:39:07.486316       1 shared_informer.go:227] caches populated
I1212 23:39:07.486334       1 shared_informer.go:204] Caches are synced for route controller 
I1212 23:39:07.486383       1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:07.486432       1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:07.486471       1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:07.486615       1 route.go:498] Finished syncing Route "example-utils/docker-registry"
E1212 23:39:07.486627       1 route.go:1308] example-utils/docker-registry failed with : can't get cert issuer: cert issuer example-utils/letsencrypt-staging is missing required secret
I1212 23:39:07.491395       1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:07.491923       1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:07.491977       1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:07.492107       1 route.go:498] Finished syncing Route "example-utils/docker-registry"
E1212 23:39:07.492122       1 route.go:1308] example-utils/docker-registry failed with : can't get cert issuer: cert issuer example-utils/letsencrypt-staging is missing required secret
I1212 23:39:07.502472       1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:07.502511       1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:07.502601       1 route.go:498] Finished syncing Route "example-utils/docker-registry"
E1212 23:39:07.502616       1 route.go:1308] example-utils/docker-registry failed with : can't get cert issuer: cert issuer example-utils/letsencrypt-staging is missing required secret
I1212 23:39:07.523170       1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:07.523218       1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:07.523366       1 route.go:498] Finished syncing Route "example-utils/docker-registry"
E1212 23:39:07.523387       1 route.go:1308] example-utils/docker-registry failed with : can't get cert issuer: cert issuer example-utils/letsencrypt-staging is missing required secret
I1212 23:39:07.563808       1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:07.563868       1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:07.564019       1 route.go:498] Finished syncing Route "example-utils/docker-registry"
E1212 23:39:07.564033       1 route.go:1308] example-utils/docker-registry failed with : can't get cert issuer: cert issuer example-utils/letsencrypt-staging is missing required secret
I1212 23:39:07.644550       1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:07.644612       1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:07.644755       1 route.go:498] Finished syncing Route "example-utils/docker-registry"
E1212 23:39:07.644771       1 route.go:1308] example-utils/docker-registry failed with : can't get cert issuer: cert issuer example-utils/letsencrypt-staging is missing required secret
I1212 23:39:07.805042       1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:07.805104       1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:07.805228       1 route.go:498] Finished syncing Route "example-utils/docker-registry"
E1212 23:39:07.805244       1 route.go:1308] example-utils/docker-registry failed with : can't get cert issuer: cert issuer example-utils/letsencrypt-staging is missing required secret
I1212 23:39:08.125389       1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:08.125466       1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:08.125633       1 route.go:498] Finished syncing Route "example-utils/docker-registry"
E1212 23:39:08.125652       1 route.go:1308] example-utils/docker-registry failed with : can't get cert issuer: cert issuer example-utils/letsencrypt-staging is missing required secret
I1212 23:39:08.765793       1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:08.765851       1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:08.765953       1 route.go:498] Finished syncing Route "example-utils/docker-registry"
E1212 23:39:08.765964       1 route.go:1308] example-utils/docker-registry failed with : can't get cert issuer: cert issuer example-utils/letsencrypt-staging is missing required secret
I1212 23:39:09.137726       1 acme.go:50] By continuing running this program you agree to the CA's Terms of Service (https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf). If you do not agree exit the program immediately!
I1212 23:39:09.451178       1 acme.go:273] Finished syncing Account "example-utils/letsencrypt-staging"
E1212 23:39:09.451200       1 acme.go:157] example-utils/letsencrypt-staging failed with : secret "letsencrypt-staging" not found
I1212 23:39:09.451259       1 event.go:281] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"example-utils", Name:"letsencrypt-staging", UID:"edcf1aa5-e5e5-4d8a-a1af-b8be8eb23acb", APIVersion:"v1", ResourceVersion:"46127840", FieldPath:""}): type: 'Normal' reason: 'AcmeAccountProvisioned' Provisioned new ACME account for issuer "example-utils/letsencrypt-staging" because its secret example-utils/letsencrypt-staging was missing.
E1212 23:39:09.453030       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"letsencrypt-staging.16c02646b2bd087c", GenerateName:"", Namespace:"example-utils", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"ConfigMap", Namespace:"example-utils", Name:"letsencrypt-staging", UID:"edcf1aa5-e5e5-4d8a-a1af-b8be8eb23acb", APIVersion:"v1", ResourceVersion:"46127840", FieldPath:""}, Reason:"AcmeAccountProvisioned", Message:"Provisioned new ACME account for issuer \"example-utils/letsencrypt-staging\" because its secret example-utils/letsencrypt-staging was missing.", Source:v1.EventSource{Component:"openshift-acme-acme-account-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc065bfe75ae4267c, ext:18287285589, loc:(*time.Location)(0x1ef3580)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc065bfe75ae4267c, ext:18287285589, loc:(*time.Location)(0x1ef3580)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events is forbidden: User "system:serviceaccount:example-utils:openshift-acme" cannot create resource "events" in API group "" in the namespace "example-utils"' (will not retry!)
I1212 23:39:09.456304       1 acme.go:271] Started syncing Account "example-utils/letsencrypt-staging"
I1212 23:39:09.891211       1 acme.go:410] Refreshed account object example-utils/letsencrypt-staging with data from ACME
I1212 23:39:09.897657       1 acme.go:193] Updating ConfigMap from example-utils/letsencrypt-staging UID=edcf1aa5-e5e5-4d8a-a1af-b8be8eb23acb RV=46127840 to example-utils/letsencrypt-staging UID=edcf1aa5-e5e5-4d8a-a1af-b8be8eb23acb,RV=46128167
I1212 23:39:09.897776       1 acme.go:273] Finished syncing Account "example-utils/letsencrypt-staging"
I1212 23:39:09.897805       1 acme.go:271] Started syncing Account "example-utils/letsencrypt-staging"
I1212 23:39:09.898269       1 acme.go:273] Finished syncing Account "example-utils/letsencrypt-staging"
I1212 23:39:10.046150       1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:10.046225       1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:10.046443       1 route.go:607] Using ACME client with DirectoryURL "https://acme-staging-v02.api.letsencrypt.org/directory"
I1212 23:39:10.646624       1 route.go:622] Created Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" for Route "example-utils/docker-registry"
I1212 23:39:10.647090       1 route.go:482] Updating status for Route example-utils/docker-registry to (*api.Status){ObservedGeneration:(int64)0 CertificateMeta:(*api.CertificateMeta)<nil> ProvisioningStatus:(api.CertProvisioningStatus){StartedAt:(time.Time)2021-12-12 23:39:10.646650622 +0000 UTC m=+19.482776539 EarliestAttemptAt:(time.Time)0001-01-01 00:00:00 +0000 UTC Failures:(int)0 OrderURI:(string)https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098 OrderStatus:(string)pending OrderError:(*api.OrderError)<nil> AccountHash:(string)} Signature:(string)}
I1212 23:39:10.660816       1 route.go:226] Updating Route example-utils/docker-registry RV=46128006->46128175 UID=9bc42f0c-30a2-4632-b587-bc890b1db48a->9bc42f0c-30a2-4632-b587-bc890b1db48a
I1212 23:39:10.660845       1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:10.661080       1 route.go:498] Finished syncing Route "example-utils/docker-registry"
I1212 23:39:10.661105       1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:10.661228       1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:10.661427       1 route.go:607] Using ACME client with DirectoryURL "https://acme-staging-v02.api.letsencrypt.org/directory"
I1212 23:39:10.666873       1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:10.666894       1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:10.672001       1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:10.677647       1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:10.677663       1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:10.688054       1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:10.693478       1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:10.693495       1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:10.713614       1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:10.719454       1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:10.719471       1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:10.759593       1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:10.764840       1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:10.764861       1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:10.844979       1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:10.850982       1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:10.851002       1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:11.011124       1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:11.017311       1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:11.017330       1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:11.254352       1 route.go:650] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" is in "pending" state
I1212 23:39:11.254380       1 route.go:655] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" contains 1 authorization(s)
I1212 23:39:11.337463       1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:11.343377       1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:11.343395       1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:11.406934       1 route.go:663] Route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": is in "pending" state
I1212 23:39:11.406967       1 route.go:690] route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": challenge "pending" is in "pending" state
I1212 23:39:11.407040       1 route.go:756] Exposer route example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang not found, creating new one.
I1212 23:39:11.425042       1 route.go:762] Created exposer Route example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang for Route example-utils/docker-registry
I1212 23:39:11.425091       1 route.go:812] Exposer secret example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang not found, creating new one.
I1212 23:39:11.429575       1 route.go:931] Exposer replica set example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang not found, creating new one.
I1212 23:39:11.441282       1 route.go:986] Exposer service example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang not found, creating new one.
I1212 23:39:11.451631       1 route.go:1008] exposer Route example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang isn't admitted yet
I1212 23:39:11.451836       1 route.go:482] Updating status for Route example-utils/docker-registry to (*api.Status){ObservedGeneration:(int64)0 CertificateMeta:(*api.CertificateMeta)<nil> ProvisioningStatus:(api.CertProvisioningStatus){StartedAt:(time.Time)2021-12-12 23:39:10.646650622 +0000 UTC EarliestAttemptAt:(time.Time)2021-12-12 23:39:10.646650622 +0000 UTC Failures:(int)0 OrderURI:(string)https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098 OrderStatus:(string)pending OrderError:(*api.OrderError)<nil> AccountHash:(string)} Signature:(string)}
I1212 23:39:11.463416       1 route.go:226] Updating Route example-utils/docker-registry RV=46128175->46128197 UID=9bc42f0c-30a2-4632-b587-bc890b1db48a->9bc42f0c-30a2-4632-b587-bc890b1db48a
I1212 23:39:11.463453       1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:11.465055       1 route.go:498] Finished syncing Route "example-utils/docker-registry"
I1212 23:39:11.465095       1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:11.465210       1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:11.465405       1 route.go:607] Using ACME client with DirectoryURL "https://acme-staging-v02.api.letsencrypt.org/directory"
I1212 23:39:11.472343       1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:11.472368       1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:11.983535       1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:11.989825       1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:11.989847       1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:12.076813       1 route.go:650] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" is in "pending" state
I1212 23:39:12.076851       1 route.go:655] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" contains 1 authorization(s)
I1212 23:39:12.233865       1 route.go:663] Route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": is in "pending" state
I1212 23:39:12.233897       1 route.go:690] route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": challenge "pending" is in "pending" state
I1212 23:39:12.234006       1 route.go:1008] exposer Route example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang isn't admitted yet
I1212 23:39:12.234231       1 route.go:498] Finished syncing Route "example-utils/docker-registry"
I1212 23:39:12.234261       1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:12.234339       1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:12.234508       1 route.go:607] Using ACME client with DirectoryURL "https://acme-staging-v02.api.letsencrypt.org/directory"
I1212 23:39:12.830891       1 route.go:650] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" is in "pending" state
I1212 23:39:12.830921       1 route.go:655] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" contains 1 authorization(s)
I1212 23:39:12.984471       1 route.go:663] Route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": is in "pending" state
I1212 23:39:12.984501       1 route.go:690] route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": challenge "pending" is in "pending" state
I1212 23:39:12.984597       1 route.go:1008] exposer Route example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang isn't admitted yet
I1212 23:39:12.984854       1 route.go:498] Finished syncing Route "example-utils/docker-registry"
I1212 23:39:14.550779       1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:14.556672       1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:14.556692       1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:14.643382       1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:14.643516       1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:14.643819       1 route.go:607] Using ACME client with DirectoryURL "https://acme-staging-v02.api.letsencrypt.org/directory"
I1212 23:39:15.249426       1 route.go:650] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" is in "pending" state
I1212 23:39:15.249463       1 route.go:655] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" contains 1 authorization(s)
I1212 23:39:15.407440       1 route.go:663] Route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": is in "pending" state
I1212 23:39:15.407469       1 route.go:690] route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": challenge "pending" is in "pending" state
I1212 23:39:15.407552       1 route.go:1008] exposer Route example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang isn't admitted yet
I1212 23:39:15.407807       1 route.go:498] Finished syncing Route "example-utils/docker-registry"
I1212 23:39:15.407833       1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:15.407909       1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:15.408026       1 route.go:607] Using ACME client with DirectoryURL "https://acme-staging-v02.api.letsencrypt.org/directory"
I1212 23:39:16.014995       1 route.go:650] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" is in "pending" state
I1212 23:39:16.015022       1 route.go:655] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" contains 1 authorization(s)
I1212 23:39:16.167121       1 route.go:663] Route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": is in "pending" state
I1212 23:39:16.167150       1 route.go:690] route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": challenge "pending" is in "pending" state
I1212 23:39:16.167236       1 route.go:1008] exposer Route example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang isn't admitted yet
I1212 23:39:16.167446       1 route.go:498] Finished syncing Route "example-utils/docker-registry"
I1212 23:39:19.676934       1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:19.682822       1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:19.682838       1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:26.451824       1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:26.451974       1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:26.452119       1 route.go:607] Using ACME client with DirectoryURL "https://acme-staging-v02.api.letsencrypt.org/directory"
I1212 23:39:27.042360       1 route.go:650] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" is in "pending" state
I1212 23:39:27.042399       1 route.go:655] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" contains 1 authorization(s)
I1212 23:39:27.200379       1 route.go:663] Route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": is in "pending" state
I1212 23:39:27.200414       1 route.go:690] route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": challenge "pending" is in "pending" state
I1212 23:39:27.200519       1 route.go:1008] exposer Route example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang isn't admitted yet
I1212 23:39:27.200794       1 route.go:498] Finished syncing Route "example-utils/docker-registry"
I1212 23:39:29.922979       1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:29.930752       1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:29.930778       1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:42.200736       1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:42.200902       1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:42.201098       1 route.go:607] Using ACME client with DirectoryURL "https://acme-staging-v02.api.letsencrypt.org/directory"
I1212 23:39:42.790856       1 route.go:650] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" is in "pending" state
I1212 23:39:42.790884       1 route.go:655] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" contains 1 authorization(s)
I1212 23:39:42.944615       1 route.go:663] Route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": is in "pending" state
I1212 23:39:42.944644       1 route.go:690] route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": challenge "pending" is in "pending" state
I1212 23:39:42.944761       1 route.go:1008] exposer Route example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang isn't admitted yet
I1212 23:39:42.945006       1 route.go:498] Finished syncing Route "example-utils/docker-registry"
I1212 23:39:50.410919       1 route.go:1171] Started syncing Route (to Secret) "example-utils/docker-registry"
I1212 23:39:50.419740       1 route.go:1173] Finished syncing Route (to Secret) "example-utils/docker-registry"
E1212 23:39:50.419757       1 route.go:1327] example-utils/docker-registry failed with : can't create Secret example-utils/docker-registry-tls: secrets "docker-registry-tls" already exists
I1212 23:39:57.944974       1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:39:57.945171       1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:39:57.945398       1 route.go:607] Using ACME client with DirectoryURL "https://acme-staging-v02.api.letsencrypt.org/directory"
I1212 23:39:58.543768       1 route.go:650] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" is in "pending" state
I1212 23:39:58.543804       1 route.go:655] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" contains 1 authorization(s)
I1212 23:39:58.701613       1 route.go:663] Route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": is in "pending" state
I1212 23:39:58.701866       1 route.go:690] route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": challenge "pending" is in "pending" state
I1212 23:39:58.701974       1 route.go:1008] exposer Route example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang isn't admitted yet
I1212 23:39:58.702202       1 route.go:498] Finished syncing Route "example-utils/docker-registry"
I1212 23:40:13.702144       1 route.go:496] Started syncing Route "example-utils/docker-registry"
I1212 23:40:13.702306       1 route.go:563] Route "example-utils/docker-registry" needs new certificate: Route is missing CertKey
I1212 23:40:13.702448       1 route.go:607] Using ACME client with DirectoryURL "https://acme-staging-v02.api.letsencrypt.org/directory"
I1212 23:40:14.290053       1 route.go:650] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" is in "pending" state
I1212 23:40:14.290086       1 route.go:655] Route "example-utils/docker-registry": Order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098" contains 1 authorization(s)
I1212 23:40:14.449247       1 route.go:663] Route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": is in "pending" state
I1212 23:40:14.449283       1 route.go:690] route "example-utils/docker-registry": order "https://acme-staging-v02.api.letsencrypt.org/acme/order/36789788/1257687098": authz "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/1164992058": challenge "pending" is in "pending" state
I1212 23:40:14.449392       1 route.go:1008] exposer Route example-utils/exposer-dsccsqnvue3fa92rf1rnf07b37p6dfdpgoqgeivoqtv8nc1buang isn't admitted yet
I1212 23:40:14.449680       1 route.go:498] Finished syncing Route "example-utils/docker-registry"

Anything else we need to know?:
I had this setup working for a few days (passthrough route + TLS secret mounted into a pod) but after recreating it in another namespace both stopped to work. No duplicate routes are present in any namespaces, double checked.
Environment:

  • OpenShift/Kubernetes version :
    Client Version: 4.8.13
    Server Version: 4.8.20
    Kubernetes Version: v1.21.4+6438632
  • Others:

@tnozicka
@ntxt ntxt added the kind/bug label Dec 13, 2021
@ntxt ntxt changed the title passthrough routes fail for exposer routes rejected for a passthrough route with a secret to mount Dec 13, 2021
@chrisegner
Copy link

chrisegner commented Jan 19, 2022 •
edited
Loading

I'm also experiencing this in the context of RH ServiceMesh 2.0 on OpenShift 4.7. As @ntxt mentioned, this used to work. We noticed certificates are expired and not renewing. I see the same stanza with the exposer route not admitted due to HostAlreadyClaimed. In my case, the offending route is the one that gets created by RHSM (istio) Gateway.

(Note: RHSM copies annotations from the Gateway to the Openshift Route object. Details https://docs.openshift.com/container-platform/4.7/service_mesh/v2x/ossm-traffic-manage.html#ossm-auto-route-annotations_routing-traffic)

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: api-gateway
  annotations:
    kubernetes.io/tls-acme: "true"
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      tls:
        mode: SIMPLE
        serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
        privateKey: /etc/istio/ingressgateway-certs/tls.key
        httpsRedirect: true
      hosts:
        - api-foo-dev.example.com
        - api-bar-dev.example.com
        - api-baz-dev.example.com
        - api-qixx-dev.example.com
        - api-quxx-dev.example.com

That causes the main route to get created in the service mesh's namespace in addition to the exposer route. These two routes conflict. If I remove the annotation, only the main route exists.

$ oc get route -n develop-istio-system
NAME                                   HOST/PORT                  PATH                                         SERVICES                            PORT    TERMINATION            WILDCARD
api-gateway-1b49626edxx                api-foo-dev.example.com                                                 istio-ingressgateway                https   passthrough/Redirect   None
api-gateway-25ac42f2exx                api-bar-dev.example.com                                                 istio-ingressgateway                https   passthrough/Redirect   None
api-gateway-2e7f2c82bxx                api-baz-dev.example.com                                                 istio-ingressgateway                https   passthrough/Redirect   None
api-gateway-7142b43bcxx                api-qixx-dev.example.com                                                istio-ingressgateway                https   passthrough/Redirect   None
api-gateway-ba7b8c932xx                api-quxx-dev.example.com                                                istio-ingressgateway                https   passthrough/Redirect   None
exposer-4dc3fbdoncopqm03ssi4khkxx      HostAlreadyClaimed         /.well-known/acme-challenge/wnq3LxaXxxxx     exposer-4dc3fbdoncopqm03ssi4khkxx   <all>   edge/Allow             None
exposer-bo90lih5l4fnd336d163ur0xx      HostAlreadyClaimed         /.well-known/acme-challenge/2tvd6HrLxxxx     exposer-bo90lih5l4fnd336d163ur0xx   <all>   edge/Allow             None
exposer-osfb5gnj37o8rr68g24a8eqxx      HostAlreadyClaimed         /.well-known/acme-challenge/_pUV-Af0xxxx     exposer-osfb5gnj37o8rr68g24a8eqxx   <all>   edge/Allow             None
exposer-ota457irmli2gpdaetqbkbfxx      HostAlreadyClaimed         /.well-known/acme-challenge/51CLcTDyxxxx     exposer-ota457irmli2gpdaetqbkbfxx   <all>   edge/Allow             None
exposer-pjqm1ebiea7nn5p41tcptt6xx      HostAlreadyClaimed         /.well-known/acme-challenge/TFXPfY8Bxxxx     exposer-pjqm1ebiea7nn5p41tcptt6xx   <all>   edge/Allow             None

Any advice on a workaround is appreciated.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@chrisegner @ntxt