You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
DynamoDb and OpenSearch mappings have potential for conflicts. One example of this being hit is with the Number type in DDB supporting both float and long types.
With dynamic mappings in OpenSearch, the first document that gets created will determine the mappings. Following documents with different types for the same field would result in mapping failures.
For example, the DynamoDb Number type can represent float, long, etc, while OpenSearch has explicit typing for float and long
Testing earlier with a field with a mix of float and long types for a field resulted in this exception sending to a serverless collection
2023-10-27T15:58:48,210 [dynamodb-pipeline-sink-worker-2-thread-1] WARN org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Document failed to write to OpenSearch with error code 400. Configure a DLQ to save failed documents. Error: mapper [cs_ship_customer_sk] cannot be changed from type [long] to [float]
2023-10-27T15:58:48,210 [dynamodb-pipeline-sink-worker-2-thread-1] WARN org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Document failed to write to OpenSearch with error code 400. Configure a DLQ to save failed documents. Error: mapper [cs_ship_customer_sk] cannot be changed from type [long] to [float]
2023-10-27T15:58:48,210 [dynamodb-pipeline-sink-worker-2-thread-1] WARN org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Document failed to write to OpenSearch with error code 400. Configure a DLQ to save failed documents. Error: mapper [cs_net_paid_inc_ship] cannot be changed from type [float] to [long]
2023-10-27T15:58:48,210 [dynamodb-pipeline-sink-worker-2-thread-1] WARN org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Document failed to write to OpenSearch with error code 400. Configure a DLQ to save failed documents. Error: mapper [cs_quantity] cannot be changed from type [long] to [float]
Describe the bug
DynamoDb and OpenSearch mappings have potential for conflicts. One example of this being hit is with the
Number
type in DDB supporting both float and long types.With dynamic mappings in OpenSearch, the first document that gets created will determine the mappings. Following documents with different types for the same field would result in mapping failures.
For example, the DynamoDb
Number
type can representfloat
,long
, etc, while OpenSearch has explicit typing for float and longTesting earlier with a field with a mix of
float
andlong
types for a field resulted in this exception sending to a serverless collectionWe need to define an index mapping template that can handle all types of DynamoDB types (https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html) to OpenSearch types (https://opensearch.org/docs/2.4/opensearch/supported-field-types/numeric/)
Expected behavior
Provide a guide for creating index mappings for ddb to opensearch. We can supply this as
template_content
.Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: