Ingress-nginx certificate handling

[nabuskey]

This is related to #127 but not specific to Backstage.

Problem:

Currently, we rely on the default TLS certificate baked in the ingress-nginx controller image to establish TLS sessions for all incoming connections. This works for accessing web UIs from browsers as long as they click on the accept risk button.

This is not so easy for applications. When we want to point an application to the https port on the ingress-nginx service, it will raise an TLS error because it's self-signed and SANs will not match.
To overcome this kind of TLS issues, many applications provide an option to skip TLS verification steps altogether. This is obviously not a good practice.

A better approach would be to provide the CA cert to applications so that they can use it to verify certs.
Unfortunately, this approach is currently impossible in idpbuilder because we rely on the default ingress-nginx certificate. Even if we extract the certificate and make it available for applications to verify against, it will fail because subject name does not match our target names (*.cnoe.localtest.me).

We need to generate our own cert, specify it in the ingress-nginx configuration, then make it available somewhere in the cluster.

Possible solutions:

Use cert manager to generate self-signed cert for ingress-nginx to use. This means we have to embed manifests like we do now for core packages. This also means we will have a pod essentially doing nothing after the initial install.

Generate it in idpbuilder CLI. Since this is a one-time operation, we could generate it in CLI for ingress-nginx to use.

[greghaynes]

Including cert management makes sense to me at a high level. A few thoughts:

• Is there a way to make a clear interface here? Cert management is something I would expect to already be in place for most orgs so its likely this part will need to be plugged out.
• Are we only thinking this would be useful for in-cluster traffic? If we also want it usable for ingress traffic to the cluster, how do we see that workflow working where the user can begin trusting the self signed cert OR they can provide a CA for use?

[jessesanford]

+1 for pluggable. As an alternate to self-signed solutions, we can use cert manager with lets encrypt, but folks will need to register their own domains and manage the challenge.

We can also use minica to stand up a local ca to generate the leaf certs and offer minica along with the rest of the CNOE stack.

Then folks would just have to make sure they include the minica root into their ca bundles, I imagine the pub root cert can be loaded into a ConfigMap or Secret for consumption.

We could sign *.cnoe.localtest.me domains without issue for any services folks using idpbuilder are developing.

Hmm it looks like we could also just use cert-manager to create a ca cert and then use that to generate leaf certs... not sure this still works. IIRC cert-manager was limited in what cert templates it would issue certs against at one point.

https://skarlso.github.io/2023/10/25/self-signed-locally-trusted-certificates-with-cert-manager/

One way or another folks consuming TLS will have to manage something out of band either DNS or their root bundle.

[nimakaviani]

This is probably one area where we should separate practices around local development from advanced use cases (e.g. deploying to external clusters).

Using minica to create a local ca and then generating leaf certs for *.cnoe.localtest.me is something we can do for local dev. This could even be done as a setup stage prior to launching the kind cluster.

For more advanced use cases, to Greg's point, I wonder whether we can assume that there is already an existing cert-manager in play that can distribute a bundle with existing CA cert etc using the trust manager?

[nabuskey]

For local clusters, we should create a self-signed CA and use that for signing. We can propagate CA cert via ESO or w/e.

As for external cluster use cases, certificate management stuff is something we shouldn't and likely not allowed to mess with.

Are we only thinking this would be useful for in-cluster traffic? If we also want it usable for ingress traffic to the cluster, how do we see that workflow working where the user can begin trusting the self signed cert OR they can provide a CA for use?

This is mostly for in-cluster traffic. I do not think we want to mess with the local machine's CA stores by ourselves. Security risks and will likely flag anti-malware stuff. We can provide an instructions to do it if the users want to do it. Using provided CA is 100% doable though.

[cmoulliard]

Different tools exist to generate a self signed certificate without using cert manager. Personally, I would recommend that cert manager becomes part of the reference implementation as it can not only generate a self certificate but if the developer, user has a domain name, then they can use let's encrypt and the ACME DNS to get a self signed and trusted CA.

Having a self signed and also trusted (by a root authority) CA is important as some browser, API, libraries, frameworks, etc will raise errors if the certificate is not trusted by a root CA authority !

FYI, I documented different approaches that I tested with gitea here: https://github.com/ch007m/my-gitea?tab=readme-ov-file#pki. The most simple is to use the gitea client and command gitea cert --host localhost -ca

[cmoulliard]

that can distribute a bundle with existing CA cert etc using the trust manager?

cert manager supports to distribute such a CA certificate: https://cert-manager.io/docs/tutorials/getting-started-with-trust-manager/#distribute-public-ca-trust ;-)