Keep a long term storage of all binaries and artifacts created by Fleet

[sharon-fdm]

Goal

User story
As a Fleet engineer,
I want to keep all versions of all our binary files
so that I can use a specific binary in case I need it.

Context

• Requestor(s): Engineering
• Product designer: _________________________

Unlike code that could be tagged and retracted with specific exact versions, binaries could not be recompiled to have the exact same file with same signature because of many reason (e.g. using different compilers).
For the reason described below and many others, it is needed to store for long term, any binary with its version.
Reason: In an incident documented here we used an existing binary with a specific SHA256 signature to remediate a problem. Similar usage may be needed in the future.

Changes

Create a mechanism that will follow our releases and store all binaries in an organized manner.
TODO: The exact organization should be discussed. (Need all files with all their versions.)

Product

• UI changes: TODO
• CLI usage changes: TODO
• REST API changes: TODO
• Permissions changes: TODO
• Outdated documentation changes: TODO
• Changes to paid features or tiers: TODO

QA

Make sure the new mechanism works on all our binaries. The new fleetd-base release mechanism stores files in new directories (stable and archive), so we do not need to worry about being backward compatible with current/old Fleet server.

Risk assessment

• Risk level: Low to none.
• Risk description: The created mechanism should have minimal load effect and should probably not have risk to override existing files.

Manual testing steps

1. Use the fleetd-base installers from:

• https://download-testing.fleetdm.com/stable/fleetd-base.pkg
• https://download-testing.fleetdm.com/stable/fleetd-base.msi
• And the 2 pointed to by https://download-testing.fleetdm.com/stable/meta.json

2. Make sure they work on Windows/macOS.

Testing notes

Confirmation

1. Engineer (@____): Added comment to user story confirming successful completion of QA.
2. QA (@getvictor): Added comment to user story confirming successful completion of QA.

[lucasmrod]

From the estimation call the following need to be addressed:

• fleetd-base.msi
• fleetd-base.pkg
• fleetd-chrome

Idea: Use a directory with version name to keep historical data (because R2 does not have versioning).

[roperzh]

@getvictor as discussed in Slack, here is (high level) what we would need on the MDM side:

1. Some way to know what's the latest released version (eg: having a /stable/ folder? or a json/xml metadata file)
2. Some way to grab the latest checksum for Windows
3. Keep the old path https://downloads.fleetdm.com/fleetd-base.msi for backwards compatibility

[roperzh]

over-communicating: I have marked this as a blocker for #19176

[getvictor]

@roperzh The updated workflow #19749 will be updating files at https://download-testing.fleetdm.com

Once this workflow is QAd, I will do another PR to switch it to https://download.fleetdm.com

[roperzh]

@xpkoala I used https://download-testing.fleetdm.com/ for the work at #20078 and can confirm that's working great.

Any way I can help you get this through ready to release?

[xpkoala]

@roperzh Nope! I was working on this with Victor yesterday and I just failed to move it over. I appreciate the extra eyes!

[roperzh]

Thanks! @getvictor as soon as you update https://download.fleetdm.com I will merge #20078

[getvictor]

@roperzh Please approve #20093

I already run the flow, so you can see the files at: https://download.fleetdm.com/stable/meta.json

[fleet-release]

Binaries stored, like seeds,
In glass city's memory,
Ensure Fleet's smooth deeds.