feat: fdb sqlite workflows driver #1850
Conversation
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
How to use the Graphite Merge QueueAdd the label merge-queue to this PR to add it to the merge queue. You must have a Graphite account in order to use the merge queue. Sign up using this link. An organization admin has enabled the Graphite Merge Queue in this repository. Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue. This stack of pull requests is managed by Graphite. Learn more about stacking. |
Deploying rivet with
|
| Latest commit: |
0ec8ed1
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://5392bbf2.rivet.pages.dev |
| Branch Preview URL: | https://01-16-feat-fdb-sqlite-workfl.rivet.pages.dev |
MasterPtato
force-pushed
the
01-16-feat_fdb_sqlite_workflows_driver
branch
9 times, most recently
from
873903d to
296889a
Compare
MasterPtato
force-pushed
the
01-16-feat_fdb_sqlite_workflows_driver
branch
from
296889a to
7171a84
Compare
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
PR Summary
This PR introduces a new FDB SQLite workflows driver for the Rivet platform. Here's a high-level summary of the key changes:
- Implements a hybrid storage solution using FoundationDB for workflow metadata and SQLite for event history tracking
- Adds comprehensive key management system for FoundationDB with proper tuple packing and serialization
- Introduces migration system for SQLite schema management with proper locking mechanisms
- Modifies workflow context and database interfaces to support the new driver alongside existing CockroachDB implementation
Key points to note:
- Adds proper transaction boundaries and retry logic for database operations
- Implements chunking capabilities for large values (though actual chunking logic is TODO)
- Includes integration tests with Docker container setup for NATS, Redis, and FDB
- Introduces new error types and handling for FDB and SQLite operations
- Adds configurable polling intervals and connection pool management
The implementation appears thorough but has some areas needing attention:
- Several TODOs around chunking implementation for large values
- Need for more specific error types instead of using anyhow::Error
- Potential performance considerations with large datasets
- Some hardcoded values in SQLite connection management
💡 (4/5) You can add custom instructions or style guidelines for the bot here!
58 file(s) reviewed, 65 comment(s)
Edit PR Review Bot Settings | Greptile
| let workflow_id = self | ||
| .ctx | ||
| .db() | ||
| .find_workflow(workflow_name, &serde_json::Value::Object(self.tags)) | ||
| .await? | ||
| .ok_or(WorkflowError::WorkflowNotFound) | ||
| .map_err(GlobalError::raw)?; |
There was a problem hiding this comment.
logic: potential race condition - workflow could be completed/removed between find and publish
MasterPtato
force-pushed
the
01-16-feat_fdb_sqlite_workflows_driver
branch
from
7171a84 to
ff931bf
Compare
MasterPtato
force-pushed
the
01-16-feat_fdb_sqlite_workflows_driver
branch
from
ff931bf to
1fbcc04
Compare
MasterPtato
changed the base branch from
main
to
01-24-fix_cluster_create_new_cluster_server_wf_with_stateful_loop
MasterPtato
force-pushed
the
01-16-feat_fdb_sqlite_workflows_driver
branch
from
1fbcc04 to
0ec8ed1
Compare
Changes