S3-Compatible Listings - Optimize and Simplify Non-Encrypted Path Support

[ferristocrat]

Summary

Storj currently encrypts object keys (paths) by default, which can lead to non-optimized listings when non-slash-terminated prefixes are used. While disabling encryption allows for lexicographically sorted listings, the process is complicated, requiring the use of Uplink CLI to generate access grants that support this behavior. Additionally, even with encryption disabled, the satellite and gateway don’t fully optimize listings, leading to non-optimal paths. This creates a poor user experience for customers expecting behavior similar to S3.

Requirements

User Story

As a Storj customer migrating from S3, I want to easily disable object key encryption at the project level so I can optimize object listings and improve performance without needing to use Uplink CLI or understand Storj’s internal mechanics.

Acceptance Criteria (WIP)

• Users can disable object key encryption at the project level directly through the Satellite Console or API.
• Object listings with unencrypted paths should follow the same performance characteristics as S3, with lexicographical sorting for non-slash-terminated prefixes.
• The Object Browser fully supports unencrypted object keys, displaying them without errors or requiring external tools.
• Clear documentation and in-app guidance are provided to users on how to manage encrypted and unencrypted paths.
  • If we go with a project setting, perhaps we could as "will you be utilizing our S3 API? Yes > we recommend disabling path encryption"

Open Item(s) for discussion

1. Is a project-level setting feasible? The goal here is to eliminate the need for customers to make a choice - a choice that can arguably be bubbled up to "do you want S3 compatibility?" if the answer is yes, then not encrypting object keys should be a prerequisite
2. Consider having this (disabling object key encryption) as the default UX when a project uses Storj managed passphrases
   • We'd need to figure out if we still want to give the option for projects with user managed passphrases (not sure what the crossover would be)

Links

• https://storj.dev/dcs/buckets/object-listings