Release 2.9.0 · opensearch-project/data-prepper

2024-08-28 Version 2.9.0

Support sets and set operations in Data Prepper expressions (#3854)
Add startsWith expression function (#4840)
Support default route option for Events that match no other route (#4615)
Delete input for processors which expand the event (#3968)
Dynamic Rule Detection (#4600)
Kafka Source should support message headers (#4565)
Aggregate processor : add option to allow raw events (#4598)
Add support for start and end times in count and histogram aggregate actions (#4614)
Add an option to count unique values of specified key(s) to CountAggregateAction (#4644)
Flatten processor: option for keys wihout brackets (#4616)
Modify Key Value processor to support string literal grouping (#4599)
Make AWS credential management available in data-prepper-config.yaml (#2570)

Support enhanced configuration of the Kafka source and buffer loggers (#4126)
Update the rename_keys and delete_entries processors to use EventKey (#4636)
Update the mutate string processors to use the EventKey. (#4649)
OpenSearch Sink add support for sending pipeline parameter in BulkRequest (#4609)
Add support for Kafka headers and timestamp in the Kafka Source (#4566)

[BUG] Visibility duplication protection fails when using S3 source for large files and receiving 10 messages from SQS queue (#4812)
[BUG] ChangeVisibilityTimeout call failure during pipeline shutdown. (#4575)
[BUG] Service-map relationship should be created regardless of missing traceGroupName (#4821)
[BUG] Unable to create stateful processors with multiple workers. (#4660)
[BUG] Routes: regex doesn't work (#4763)
[BUG] Grok plugin CLOUDFRONT_ACCESS_LOG pattern does not compile (#4604)
[BUG] The user_agent processor throws exceptions with multiple threads. (#4618)
[BUG] DynamoDB source export converts Numbers ending in 0 to scientific notation (#3840)
Fix null document in DLQ object (#4814)
Fix KeyValue Processor value grouping bug (#4606)