Recover from Fastly errors

[anothertobi]

Context

We recently got warnings for expiring certificates on a few pages. The TLS secrets for these pages
had the label fastly.amazee.io/paused set to true. with the paused reason being a 503 error from Fastly. This prevented the controller from uploading the certificates updated by cert-manager.

Example:

kind: Secret
metadata:
  annotations:
    fastly.amazee.io/paused-at: "2023-03-26 22:51:54"
    fastly.amazee.io/paused-reason: |
      Unable to get certificate information from Fastly, error was: 503 - Service Unavailable:

Manually cleaning up after Fastly had an internal error is unnecessary toil.

Ideas

• Requeue secrets after 503 error from Fastly
• Requeue secrets after any error from Fastly
• Requeue secrets after any error -> My personal favorite as this does not require manual intervention and makes use of the built-in exponential backoff to not overload any (external) APIs.

Alternatives

The label fastly.amazee.io/paused must be set to false or removed manually like: kubectl -n prod label secret amazee.io-tls fastly.amazee.io/paused-.