Eventing/ Event Notifications

[ferristocrat]

Background/ Summary

When files land in the auto ingest bucket we get a notification to our api, we then pick up these assets and bring them into our platform and begin to process them as fast as we can. This allows users to drop large amounts of assets on us in a fire-and-forget fashion, either sent from their own system or manually using tools like cyberduck.
All we need is an api notification once the file finishes being written to Storj so we can go and fetch it. Other notification for other actions (delete, replace etc) are helpful but nowhere near as important.

What is the problem/pain point?

Customers and developers utilizing Storj for file storage currently lack a mechanism to receive real-time notifications about bucket changes. This limitation can hinder the automation of certain workflows and processes that rely on timely data updates.

Who is impacted?

• Customers building applications on top of Storj, especially those migrating or comparing with AWS S3.

What is the impact?

Without event-driven notifications:

1. Customers face challenges in automating workflows that depend on bucket changes.
2. Customers need to spend additional resources/effort on developing workarounds.
3. Storj might be perceived as less feature-rich compared to competitors

Why now?

1. Competitors like AWS already offer this feature, and others like Wasabi are on track to introduce it.
2. Clear demand from customers, as indicated by their feedback.
3. Enhancing Storj's compatibility with S3 methods will further establish its reputation as a top S3-compatible storage solution.

Requirements

User Story

As a developer using Storj for object storage, I want to receive notifications about bucket changes so I can automate workflows and respond to data changes in a timely manner.

Acceptance Criteria

1. Users should be able to set up notifications for the events listed in the table below.
2. Users should be able to register/subscribe to specific event-driven notifications.
3. Notifications should be delivered in real-time or near-real-time following a bucket change event.
4. Comprehensive documentation on how to configure and utilize this feature should be available.
5. [Nice to have] The following S3 methods should be supported:
   • GetBucketNotificationConfiguration
   • PutBucketNotificationConfiguration

Event	Description
Object Created	New object added to bucket.
Object Deleted	Object deleted from bucket.
Object Updated	Object changed (or new version created)
Versioning related events	TBD on if/what types of events related to versioning we will have
Object Copied	TBD if needed: Server side copy performed using existing object.

Success Metrics

1. Positive feedback from users who requested this feature.
2. An uptick in new customers or retention of existing ones due to this enhanced functionality.
3. Growing use of feature after launch

---

References:

• Amazon S3 Event Notifications
• GetBucketNotificationConfiguration
• PutBucketNotificationConfiguration

[moinologics]

When developer can expect this feature to be released? Any timeline

[ferristocrat]

When developer can expect this feature to be released? Any timeline

Hey @moinologics - still in the "idea" stage... hasn't really reached critical mass in terms of requests for this vs other features. Happy to chat more about your use case/needs as well!