Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] DynamoDB source has potential for invalid mappings to OpenSearch #3557

Closed
graytaylor0 opened this issue Oct 27, 2023 · 1 comment
Closed
Labels
bug Something isn't working

Comments

@graytaylor0
Copy link
Member

graytaylor0 commented Oct 27, 2023

Describe the bug
DynamoDb and OpenSearch mappings have potential for conflicts. One example of this being hit is with the Number type in DDB supporting both float and long types.

With dynamic mappings in OpenSearch, the first document that gets created will determine the mappings. Following documents with different types for the same field would result in mapping failures.

For example, the DynamoDb Number type can represent float, long, etc, while OpenSearch has explicit typing for float and long

Testing earlier with a field with a mix of float and long types for a field resulted in this exception sending to a serverless collection

2023-10-27T15:58:48,210 [dynamodb-pipeline-sink-worker-2-thread-1] WARN  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Document failed to write to OpenSearch with error code 400. Configure a DLQ to save failed documents. Error: mapper [cs_ship_customer_sk] cannot be changed from type [long] to [float]
2023-10-27T15:58:48,210 [dynamodb-pipeline-sink-worker-2-thread-1] WARN  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Document failed to write to OpenSearch with error code 400. Configure a DLQ to save failed documents. Error: mapper [cs_ship_customer_sk] cannot be changed from type [long] to [float]
2023-10-27T15:58:48,210 [dynamodb-pipeline-sink-worker-2-thread-1] WARN  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Document failed to write to OpenSearch with error code 400. Configure a DLQ to save failed documents. Error: mapper [cs_net_paid_inc_ship] cannot be changed from type [float] to [long]
2023-10-27T15:58:48,210 [dynamodb-pipeline-sink-worker-2-thread-1] WARN  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Document failed to write to OpenSearch with error code 400. Configure a DLQ to save failed documents. Error: mapper [cs_quantity] cannot be changed from type [long] to [float]

We need to define an index mapping template that can handle all types of DynamoDB types (https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html) to OpenSearch types (https://opensearch.org/docs/2.4/opensearch/supported-field-types/numeric/)

Expected behavior
Provide a guide for creating index mappings for ddb to opensearch. We can supply this as template_content.

Additional context
Add any other context about the problem here.

@christopheranderson
Copy link

Could we include primary key info in the error context?

@github-project-automation github-project-automation bot moved this from Unplanned to Done in Data Prepper Tracking Board Jan 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Development

No branches or pull requests

2 participants