Disaster / Recovery when Kargo gets deleted

[jkleinlercher]

How do I recover the current state of the pipelines when Kargo got uninstalled and reinstalled? Can Kargo recover the state by itself?

[krancour]

Sorry for the delayed response. This has been much discussed internally lately. It's a bit of good news / bad news.

Starting with what happens if you somehow lose all your Kargo resources:

If you're doing real gitops and not taking shortcuts, like updating the state of Argo CD Apps without writing back to Git, then losing all Kargo resources would be inconvenient, but would have no effect on the health of your environments.

Argo CD Apps (assuming they weren't in the same cluster as the Kargo resources that were lost) would also continue to function correctly, happily syncing to whatever it is they sync to, e.g. some path in the head of some branch.

If you then re-created your Kargo resources, you'd get all your pipelines back, but they'd look empty. Warehouses would immediately begin discovering artifacts and producing Freight. The re-created "empty" pipelines would be immediately ready to move that new Freight from left to right and and as that happens, things would start to look normal again.

What you would have lost that cannot be recovered is all your old Freight and the (limited) history that's stored in each Stage's status. If this is a concern, we recommend backing up the Kargo control plane's cluster with something like Velero or Kasten.

Can Kargo recover the state by itself?

In an ideal world, it would, but there are at least a couple things standing in the way:

1. Freight are "cattle; not pets." You probably GitOps your Projects and Stages and Warehouses, but Freight are (for the most part) system-created (and occasionally user-created) and eventually garbage collected after they've fallen out of use for an extended period of time. Short of backing them up to some other datastore (something we don't currently intend on doing) there would be no way to repopulate the "Freight timeline" with everything that was there before.

2. Kargo has become progressively less opinionated over time, which has led us to a place where promotion processes are user-defined. There's no realistic possibility of reverse engineering (often complex) user-defined processes into an understanding of the current state for the world. This problem gets compounded when we introduce support for custom / third-party promotion steps.

Coming back to this:

we recommend backing up the Kargo control plane's cluster with something like Velero or Kasten.

I tend to avoid stumping for Akuity's Kargo SaaS here on GitHub, but it's worth saying that backup of Akuity hosted Kargo (and Argo CD) control plane clusters is a feature baked into our platform at a lower level.