2.5.0

2023-10-09 Version 2.5.0

---

Features

• Support OpenSearch as source. (#1985)
• Support translate processor. (#1914)
• Support dissect processor. (#3362)
• Support AWS secrets in pipeline and Data Prepper config as an experimental feature. (#2780)

Enhancements

• Support update, upsert, delete bulk actions in OpenSearch sink. (#3109)
• Support inline index templates in OpenSearch sink. (#3365)
• Add retry to Kafka consumer in source. (#3399)
• Support OpenTelemetry SeverityText for logs. (#3280)
• Merging PipelineDataflowModel instead of pipeline YAML files. ([#3289]#3289)
• Support recursive feature in KeyValue processor. (#888)

Bug Fixes

• Fix NullPointerException in S3 scan when bucket kay has null value. (#3316)
• Fix a bug where S3 source does not stop on pipeline shutdown. (#3341)
• Fix exemplar list in Histogram and Count aggregations. (#3364)

Security

• Fix CVE-2023-44487, HTTP/2 reset floods. (#3474)
• Fix CVE-2023-4586. (#3443)
• Fix CVE-2023-39410. (#3430)

Maintenance

• Build with Gradle 8. (#3287)
• Remove sleep from Kafka source timeout test. (#3263)
• Enable Gatling HTTPS support and path configuration. (#3308)
• Support Gatling tests using AWS sigV4 signing. (#3311)
• Support local ARM image build. (#3352)

2.4.1

2023-10-02 Version 2.4.1

---

Enhancements

• Add support for fully async acknowledgments in source coordination. (#3391)

Bug Fixes

• Fix NullPointerException in S3 scan partition supplier. (#3323)
• Fix issue caused by InterruptedException in S3 source where source is polling after pipeline shutdown. (#3345)
• Update trace analytics sample app to run again. (#3353)

Maintenance

• Improve logging for failed documents in the OpenSearch sink. (#3389)
• Update common-codec to 1.16.0 (#3370)
• Update hibernate-validator to 8.0.1.Final. (#3369)
• Update trace analytics sample app to run using the latest Spring Boot 3.1.3. (#3346)
• Update Gradle to 8.3. (#3300)
• Update grpcio from 1.50.0 to 1.53.0. (#3315)
• Update certifi from 2022.12.7 to 2023.7.22. (#3314)
• Update Bouncy Castle to 1.76. (#3307)
• Reduce sleep times in BlockingBufferTests to speed up unit tests. (#3287)
• Update checkstyle dependency to 10.12.3. (#3286)
• Remove Maxmind license keys from test URLs. (#3285)
• Remove unnecessary dependencies in the S3 sink and Parquet codecs. (#3283)

Security

• Update armeria to 1.25.2 to fix CVE-2023-32732, CVE-2023-38493. (#3366)
• Fix CVE fixes CVE-2022-36944, WS-2023-0116, CVE-2021-39194, CVE-2023-3635, CVE-2023-36479, CVE-2023-40167. (#3392)
• Update commons-compress to 1.24.0 to fix CVE-2023-42503. (#3388)
• Fix CVE-2022-45688, CVE-2023-43642. (#3409)

2.4.0

2023-08-28 Version 2.4.0

---

Features

• Support Kafka as source. (#254)
• Support source coordination. (#2412)
• Support S3 scan capability in S3 source. (#1970)
• Support ElasticSearch 6.8 in OpenSearch sink. (#3003)
• Support custom index template in OpenSearch sink with ElasticSearch 6.8. (#3060)
• Support filtering data in sink using include_keys and exclude_keys. (#2975)

Enhancements

• Support generic sink codec structure for sinks. (#2403)
• Support expressions in OpenSearch index and document ID. (#2864)
• Support defining bucket ownership. (#2012)
• Add exemplars to metrics generated in aggregate processor. (#3164)
• Add cardinality key support in Anomaly detector processor using identification_keys. (#3073)
• Support allow_duplicate_values in key-value processor. (#889)
• Support remove_brackets in key-value processor. (#892)
• Support exclude_keys in key-value processor. (#890)
• Support default_keys in key-value processor. (#891)
• Add GZip compression in S3 sink. (#3130)
• Add Snappy compression in S3 sink. (#3154)
• Update metric for ECF instance from counter to gauge and fix flaky tests caused by RCF variance. (#3145)
• Support s3:// prefix in pipeline where s3 bucket names are used. (#3143)
• Update circuit breaker configuration log message. (#3175)
• Deprecate document_id_field in support of document_id. (#3074)

Bug Fixes

• Fix race condition in sources using End-to-End Acknowledgments. (#3038)
• Fix DLQ deserialization with create action. (#3040)
• Fix IllegalArgumentException in csv processor when key does not exist. (#3053)
• Handle RequestTimeoutException in push based sources correctly. (#3063)
• Fix S3 sink writing to closed stream. (#3160)
• Fix timestamp used in S3 sink to 24-hour format. (#3171)
• Fix stale buffer data not being written to S3 sink. (#3187)
• Fix IllegalArgumentException in convert entry processor. (#3135)
• Fix ClassCastException in parse_json processor with OTel logs source. (#3184)
• Fix UnexpectedTypeException in S3 select using NotBlank annotation. (#3208)

Maintenance

• Update existing release workflow to trigger Data Prepper release jenkins job. (#2122)
• Reduce test time which reduces build time. (#3019, #3020, #3021)
• Updated GitHub actions to use Data Prepper in job titles. (#3104)
• Add Apache commons-lang3 to gradle catalog. (#3120)
• Add integration test coverage for ODFE 0.10.0 OpenSearch sink. (#3131)
• Updated documentation process in developer guide. (#2772)
• Fix flaky conditional routing test. (#3139)
• Fix flaky unit tests. (#3150)
• Add integration test for S3 sink. (#3179)
• Update Data Prepper tar.gz to include JDK 17.0.8+7. (#3136)
• Update S3 sink to speed up unit test time. (#3203)

2.3.2

2023-07-12 Version 2.3.2

---

Bug Fixes

• Updated the release date (#2912)
• Fix addTags API in EventMetadata (#2996)
• Fix DLQ writer writing empty list (#2998)
• Fix SqsWorker error messages (#3002)
• Fix S3 errors around end of file behavior. (#3006)
• Retry s3 reads on socket exceptions. (#3008)
• Fix race condition in SqsWorker when acknowledgements are enabled (#3010)
• Remove validation that made keys starting or ending with . - or _ invalid (#3007)

Security

• Fix CVE-2023-35165, CVE-2023-34455, CVE-2023-34453, CVE-2023-34454, CVE-2023-2976 (#2952)
• Fix bucket ownership validation. This was a regression introduced in Data Prepper 2.3. (#3011)

2.3.1

2023-06-20 Version 2.3.1

---

Enhancements

• Add support for external ID when making STS AssumeRole call (#2862)
• Remove sensitive from the error log for index name format failure in OpenSearch sink (#2894)

Bug Fixes

• Fix errors when SQS notifications are from AWS EventBridge (#2861, #2789, #2788)
• Fix concurrentModification in CredentialsCache (#2876)
• Fix suppressed exception and added logs when incorrect compression is configures in S3 source (#2879, #2896)
• Fix S3 sink metrics names as they are conflicting with S3 source (#2887)
• Fix silent dropping data when index has null keys (#2885)

Security

• Fix CVE-2017-1000487, CVE-2022-4244, CVE-2022-4245 (#2848)

2.3.0

2023-06-06 Version 2.3.0

---

Features

• Supports S3 as a sink. (#1048, #2732)
• Tagging events. (#629)
• Adds a function for getting the length of a string using Data Prepper Expressions. (#2639)
• Adds a function to determine if an IP address is in a CIDR block. (#2625)
• Support for expressions to extract string or numerical values in Data Prepper Expressions. (#2686, #2685)
• Read Parquet files using a codec for the S3 source and other codec-compatible sources. (#2445)
• Read Avro files using a codec for the S3 source and other codec-compatible sources. (#2527)
• Adds an obfuscate processor to help obfuscate data. (#1952)
• Support parsing User-Agent headers with a user_agent processor. (#2618)
• Support expressions in add_entries processor. (#2672)
• Support adding event data from event metadata in add_entries processor. (#2687)
• Adds a tail sampling processor. (#2572)
• Adds when conditions for many difference processors. (#2613)

Enhancements

• Support OpenSearch composable index templates. (#1275)
• Adds input codec plugin concept to Data Prepper. (#1532)
• Improves error messages on authorization failures with OpenSearch clusters. (#2657, #2655)
• Clearer metrics when S3 source does not detect events in S3 objects. (#2699)
• Allow override the Log4j configuration file when running Data Prepper. (#2720)
• Pipeline creation succeeds even when sinks are not ready. (#2656)
• Support compression in OTel sources. (#1176)
• Support gzip compression in HTTP source. (#1399)
• Support functions in Data Prepper Expression language. (#2626)
• Support SNS to SQS messages in S3 source. (#2604)
• Support shared AWS credentials across pipeline components. (#2751, #2764)
• Initial extension plugin support. (#2636, #2637)

Bug Fixes

• Improves how the opensearch sink retries requests to OpenSearch with a maximum wait time. (#2641)
• Improves how the opensearch sink initializes by continuing to retry failure requests. (#2769)
• Fixes an exception when writing to OpenSearch. (#2654)
• Improves error logs for document failures in opensearch sink. (#2597)
• Fixes issues with S3 key prefixes in the opensearch S3 DLQ. (#2581)
• Updates Random Cut Forest dependency to reduce noise. (#2783)
• Fixes to writing to the Data Prepper buffer in the S3 source. (#2701)
• Fixes S3 source backoff which was too aggressive in the requests it made. (#2568)

Security

• CVE-2021-22096 (#2671)
• CVE-2021-22060 (#2670)

Maintenance

• DataPrepper ExpressionEvaluator should not be type specific. (#2703)
• Change JsonStringBuilder in JacksonEvent to be non static for ease-of-use. (#2665)

2.2.1

2023-05-09 Version 2.2.1

---

Enhancements

• Log full errors when OpenSearch sink fails to start (#2565)
• Increase backoff delays in S3 source (#2568)
• Add object filter patterns for Core Peer Forwarder (#2576)
• Improve logging when OpenSearch sink fails to push (#2597)
• Update OpenSearch sink retry mechanism (#2643)

Bug Fixes

• Fix NullPointerException in OpenSearch sink for bulkRetryCountMap (#2600)

Security

• Fix CVE-2022-1471 (#2632)
• Fix CVE-2023-20863, CVE-2022-45143 (#2632)
• Fix CVE-2022-41854, CVE-2022-1471 (#2651)

Maintenance

• Update kotlin-stdlib, kotlin-stdlib-common to 1.8.20 (#2434)
• Update log4j-jpl to 2.20.0 (#2430)
• Update byte-buddy-agent to 1.14.3 (#2430)
• Cleanup OpenSearch sink, OTel trace processor gradle files and use Gradle catalog (#2569)
• Fix End-to-End acknowledgement integration test (#2551)

2.2.0

2023-04-20 Version 2.2.0

---

Features

• Support End-to-End Acknowledgments for S3 source (#2394, #2426, #2436, #2458, #2465)
• Support S3 DLQ for OpenSearch sink (#2298)
• Support Amazon Opensearch Serverless Sink (#2169)
• Support S3 Select in s3 source (#1971)
• Support new list_to_map processor (#2410)
• Support Snappy compression in s3 source (#2420)
• Add append aggregation action for aggregate processor (#2230)
• Support built-in grok patterns for common use-cases (#2514)

Enhancements

• Improve logging by adding error details to logs (#2391, #2417, #2555, #2448, #2438)
• Add EventFactory framework to create events (#2378, #2399 #2426)
• Support routes as an alias to route component (#2550)
• Update Java serialization allowlist to have specific classes for JsonNode (#2376)
• Support configurable retries in OpenSearch sink (#2339)
• Use AwsSdk2Transport with OpenSearch Java client (#1881)
• Add sqsMessagesDeleteFailed to track SQS delete failures (#2450)
• Retry TimeoutException caused by flushing data to buffer in S3 source (#2470)
• Add IAM role validation to S3 source and OpenSearch sink (#2472)
• Allow deprecated name for plugins (#2504)
• Make shutdown behaviour configurable (#2524)
• Add support for document_root_key to configure the key to write to OpenSearch (#2511)
• Support format option in add_entries processor (#2464)
• Support isolated service map extraction in service-map (#628)

Bug Fixes

• Fix missing data points in EMFLoggingMeterRegistry (#2469)
• Fix Trace analytics sample app requirement dependencies (#2477)
• Shutdown log_generator properly (#2494)
• Fix CSV codec drops data without metrics (#2512)
• Fix bug where Java process doesn't get killed when Data Prepper terminates (#2441)

Security

• Fix CVE-2023-20863, CVE-2023-1370, WS-2019-0379 (#2546)

Maintenance

• Update package name for otel_logs_source from com.amazon to org.opensearch (#2543)
• Updated the following plugin names, otel_trace_raw to otel_traces, otel_metrics_raw_processor to otel_metrics, service_map_stateful to service_map (#2526)
• Move backoff and retry logic to BufferAccumulator (#2481)
• Remove unnecessary calls in file sink. (#2475)
• Clean up un-used Peer Forwarder processor gradle file (#2386)
• Fix Java doc warning (#2396)
• Use Gradle version catalogs for dependency management (#2382)
• Update developer guide links to Documentation (#2367)
• Update log4j-bom to 2.20.0 (#2334)
• Update commons-io to 2.11.0 (#2335, #2336)
• Update byte-buddy to 1.14.2 (#2407)
• Update randomcutforest to 3.6.0 (#2507)
• Update micrometer-bom to 1.10.5 (#2433)
• Replace Grok debugger in documentation (#2425)
• Test against OpenSearch 2.6.0 version (#2455)

2.1.1

2023-03-16 Version 2.1.1

---

Bug Fixes

• Removed traceState from Link as it is not required (#2363)
• Fix IllegalArgumentException in PluginMetrics when null value for pipeline name is passed from OTel sources (#2369)

Maintenance

• Removed constraints on Netty and use version supplied by dependencies (#2031)

2.1.0

2023-03-02 Version 2.1.0

---

Features

• Support OpenTelementry Logs (#1372)
• New type conversion processor (#2010)
• New anomaly detection processor (#2040)
• Support histogram action in aggregate processor (#2078)
• Support percent sampler action in aggregate processor (#2094)
• Support rate limiter action in aggregate processor (#2088)
• Provide a circuit breaking capability (#2150)

Enhancements

• Support custom headers when assuming AWS IAM roles (#1888)
• Provide additional metrics for S3 source (#2025)
• Support "when" condition in aggregate processor actions (#2021)
• Support dynamic index name in OpenSearch sink (#1459)
• Support latest OTEL metrics features (#1310)
• Support loading files from S3 in OpenSearch sink (#2120)
• Support metric tags based on metric name pattern (#2229)
• Support retry when sink fails to initialize (#936)
• Add more metrics for OpenSearch sink (#2168)
• Validate file content from S3 for OpenSearch sink (#2178)
• Add more configurable values to OTEL trace raw processor (#2165)
• Add more metrics for OTEL trace raw processor (#2166)
• Provide logging markers to filter sensitive data and events (#1990)
• Support sensitive data masking in logs (#2187)
• Emit Metrics for ACM and AWS CloudMap related calls (#2087)
• Add buffer overflow metric in AbstractBuffer (#2167)
• Support configurable read batch delay for peer-forwarder (#2158)
• Improve ReceiveBuffer Performance (#2142)
• Support OTEL scope logs (#2115)
• Implement batching for peer forwarder request documents (#2197)
• Add more metrics to OTEL metrics source (#2283)
• Support a version property in pipeline YAML configurations (#2263)
• Change the path prefix for OTEL endpoints (#2257)
• Change the path for HTTP source (#2258)
• Improve peer-forwarding serialization (#2242)
• Add null support to Data Prepper Expressions (#1136)
• Support for complex document ID and routing ID (#1966)
• Add PluginMetrics in the Auth Plugin for HTTP, OTEL and Metrics Source (#2023)
• Support Long type in expressions (#2265)
• Add backoff for SQS in S3 source to reduce logging (#2326)
• Add several performance improvement changes (#2190, #2228, #2238, #2239, #2241, #2246, #2250, #2290)

Bug Fixes

• Fix logging by updating related packages for SLF4J 2.0 (#2113)
• Fix null pointer exception in remote peer forwarding (#2124)
• Fix Data Prepper to not terminate on invalid open telemetry metric/trace data (#2176)
• Fix Armeria exception when under pressure (#2274)
• Fix release build issue (#2345)

Security

• Fix CVE-2022-3509, CVE-2022-3510 (#2079)
• Fix CVE-2022-41881, CVE-2021-21290 and CVE-2022-41915 (#2093)
• Fix CVE-2020-8908, CVE-2018-10237 (#2254)
• Update OpenTelemetry dependencies to address CVE-2023-25577 and CVE-2023-23934 (#2288)
• Use secure Java serialization with Java serialization (#2310)

Maintenance

• Test against multiple OTEL version (#1963)
• Document trace_peer_forwarder in README (#2141)
• Use consistent AWS pipeline configurations (#2184)
• Use snake case for all configurations (#2203)