Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support AWS Aurora/RDS PostgreSQL as source #5309

Open
9 tasks
oeyh opened this issue Jan 6, 2025 · 0 comments
Open
9 tasks

Support AWS Aurora/RDS PostgreSQL as source #5309

oeyh opened this issue Jan 6, 2025 · 0 comments
Labels
enhancement New feature or request Roadmap:Search Project-wide roadmap label

Comments

@oeyh
Copy link
Collaborator

oeyh commented Jan 6, 2025

Is your feature request related to a problem? Please describe.
As a Data Prepper user, I would like to have an rds source to load existing data and stream change events from RDS MySQL databases.

Describe the solution you'd like
For export (loading existing data), we can create a snapshot, export it to S3 and read the data from S3
For stream (streaming change events), we can create a Postgres logical replication stream to receive change events.

Describe alternatives you've considered (Optional)
Run SQL queries periodically through a JDBC driver to load existing and incremental data from the source database.

Additional context
Relevant issue for MySQL: #4561

Tasks

  • Export implementation
  • Stream implementation
  • Checkpointing in both export and stream
  • Secret rotation support
  • Add E2E acknowledge support
  • Add data type mapping
  • Add plugin metrics
  • Add aggregate metrics
  • Add integration tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Roadmap:Search Project-wide roadmap label
Projects
Status: 2.12 (Jan/Feb 2024)
Status: New
Development

No branches or pull requests

1 participant