[10 Jan 2025] - Secret-based key rotation

[BlaineEXE]

Enhancement

Is your feature request related to a problem?/Why is this needed

Currently, key rotation is allowed only with IAM-style authentication. Consider benefits and implementation of secret-based rotation.

Describe the solution you'd like in detail

Key rotation is a critical part of secure infrastructure. Key rotation should be fairly
straightforward with IAM-style auth, but what do we do about KEY auth?

Blaine's proposal:
Users can theoretically update keys in 2 places:

• the secret containing KEY auth info
• directly in the OSP backend

If users change keys directly in the OSP backend, it will be difficult for COSI to receive an event
that causes it to reconcile and update necessary resources. In this case, a user's best workaround
may be to restart the COSI controller and/or OSP driver to re-reconcile all bucketaccesses, any of
which may have been updated. COSI can then update the key secret with the latest info.

COSI could allow users to modify the secret (even allowing users to pre-populate the secret) with
auth information. From discussions with Rook users, many use a service to automatically rotate
auth data in k8s Secrets periodically.

If COSI does this, it will have to account for some corner cases, especially for S3 protocols.

In S3, users may want to prepopulate both access key and secret key, but AWS's spec for S3 (to the best of my current knowledge) always randomizes access keys. COSI devs would have to consider corner cases like this one during development.

Describe alternatives you've considered

COSI can continue to support key rotation only for IAM-style authentication.

Additional context
N/A