What is the correct way to configure cross-cluster communication and analysisRuns?

[pww217]

There are a few issues and discussions around this but there's no clear answer that I can see.

My end goal is to run analysisRuns on the remote clusters where workloads actually run, not the mgmt cluster that the main Kargo controller runs on.

I gathered from other threads and docs that the ArgoCD model of basically putting a KubeConfig in a secret and calling across clusters in that fashion is frowned up.

But the alternative or best practice config is not really documented as far as I can tell. It seems to be something like: set up a Kargo controller on each cluster with one as the primary, then use some sort of communication/webhooks between them to run remote AnalysisRuns and do other operations on a remote cluster.

Basically Kargo <--> Kargo communication instead of authing with a cluster directly, if I understand right.

That sounds great to me but I really can't find how to set it up much of anywhere. Looking through the docs, issues, discussions, and helm chart documentation it's really not very clear to me what the best practice is to do this.

It's important to me because it's really the final thing I need to prove that Kargo is capable of internally in order to sort of begin actually using it more widely. If we want to replace GHA we need to bring the ability to run arbitrary code in between promotions (unit, E2E, integration testing) and it seems that Kargo is perfectly capable of this, it's just a bit hard to figure out how to do it in remote clusters.

I think this will be a common use case going forward as being able to run analysisTemplates in the same cluster/namespace as the actual workload of course makes for more accurate, reliable, and simple (less complex networking and such) testing.

So any guidance, docs, issues, or anything else folks could point me to would help a lot! A helm example in particular would be most valuable.

[krancour]

This isn't documented yet, but it's basically this:

1. When installing a distributed (i.e. sharded) Kargo controller somewhere besides the cluster that actually hosts the Kargo control plane, provide a cluster-specific value for controller.rollouts.controllerInstanceID during the helm install.

2. Run a Rollouts controller "next to" each of your distributed Kargo controllers. Following process quite similar to how you must have already configured the distributed Kargo controllers, configure it like so:

   1. Create a Secret with kubeconfig for connecting back to the Kargo control plane's cluster.
   2. Mount that Secret somewhere in the controller's container. Assuming you install with Helm, use controller.volumes and controller.volumeMounts for this.
   3. Point the controller at the kubeconfig using the --kubeconfig flag. Assuming you install with Helm, use controller.extraArgs for this.
   4. Also start the controller with --instance-id set to the cluster-specific value mentioned in the first step. Assuming you install with Helm, use controller.extraArgs for this as well.

What you'll end up with is:

• Both the distributed Kargo controller and its nearby Rollouts contoller connected to the Kargo control plane.
• That controller will create AnalysisTemplates and Analysis runs in the Kargo control plane and apply an Rollouts instance ID label to all of them.
• The rollouts controller will watch/reconcile Rollouts resources in the Kargo control plane, but only those labeled with a certain instance ID.

[krancour]

Note this will not conflict with any other Rollouts controller that might happen to be running in that cluster.

[krancour]

I forgot to mention the most important part. I'm pretty sure you knew this anyway, but for the benefit of anyone else who reads this thread in the future, any Jobs will run where the Rollouts controller is running. 😄

[pww217]

Appreciate the quick and detailed response as always!

I modified an existing diagram I have to try to visualizing this architecture. I have a lot of trouble with this stuff without diagramming it out, and maybe it'll also help future lurkers.

I don't think it's perfectly accurate but I tried to make all the connections as I understand them. Is this basically accurate or what would you change? It's a bit simplified/has generic names but hopefully it drives home the key ideas.

[pww217]

I think one thing about this diagram is that ideally the rollout/job would run in the app's namespace itself in ideal conditions, to match the runtime environment as closely as possible. Traditionally we've mirrored this as best as we can with cross-cluster GHA runners and similar but obviously it's simpler, more reliable, and more accurate to run verifications alongside the real workload (or as close as feasible). Interested in any thoughts on that, too.

[krancour]

This looks close. I'd say your understanding is fairly good.

I'd put boxes in the clusters on the righthand side to depict the rollouts controllers.

I gather you have one Argo CD control plane in your tooling cluster and it's managing Apps in both of the other clusters. People do do this, but just a heads up that this is generally thought of to be less secure than having an Argo CD control plane per cluster.

The reason is that your one Argo CD control plane has highly privileged credentials for multiple clusters. It becomes a very high value target. Kargo avoided this by distributing the controllers to the different clusters and having them "phone home" to the control plane.

[liron-smarsh]

Hi @krancour , that did the work and now AnalisysysRun's works from the remote clusters thank you so much for that much needed explanation!
I"m hitting another issue here, the issue is that on on Kargo UI I"m no longer able to view AnalisysysRun's output anymore it just stopped responding even though it can tell that it was successful (via the UI) there is no verification presentation ( I was able to verify that the analysis run gets created in to control plane cluster)
referring to this window that is expected to work if I was not clear enough:

Also getting this console error when clicking the Analysis run:

[krancour]

It looks like it's choking on a response body that was supposed to be JSON but wasn't. Presumably, that's the response that should have contained the logs you're looking for. Would you be able to use the browser's developer tools to find the response in question? It's possible that the response you're receiving isn't from the API server, but is an error response of some kind from a proxy between you and the API server. That's the most likely reason I can think of that the response would not be JSON as expected.