You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
As a Data Prepper user, I would like to have an rds source to load existing data and stream change events from RDS MySQL databases.
Describe the solution you'd like
For export (loading existing data), we can create a snapshot, export it to S3 and read the data from S3
For stream (streaming change events), we can create a Postgres logical replication stream to receive change events.
Describe alternatives you've considered (Optional)
Run SQL queries periodically through a JDBC driver to load existing and incremental data from the source database.
Additional context
Relevant issue for MySQL: #4561
Tasks
Export implementation
Stream implementation
Checkpointing in both export and stream
Secret rotation support
Add E2E acknowledge support
Add data type mapping
Add plugin metrics
Add aggregate metrics
Add integration tests
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
As a Data Prepper user, I would like to have an rds source to load existing data and stream change events from RDS MySQL databases.
Describe the solution you'd like
For export (loading existing data), we can create a snapshot, export it to S3 and read the data from S3
For stream (streaming change events), we can create a Postgres logical replication stream to receive change events.
Describe alternatives you've considered (Optional)
Run SQL queries periodically through a JDBC driver to load existing and incremental data from the source database.
Additional context
Relevant issue for MySQL: #4561
Tasks
The text was updated successfully, but these errors were encountered: