Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When will Production ready Http Sink be included in distribution? #5270

Open
RashmiRam opened this issue Dec 17, 2024 · 4 comments
Open

When will Production ready Http Sink be included in distribution? #5270

RashmiRam opened this issue Dec 17, 2024 · 4 comments
Labels
enhancement New feature or request

Comments

@RashmiRam
Copy link

Is your feature request related to a problem? Please describe.
We are planning to leverage HTTP sink to index the records to Clickhouse. I see the http sink is not enabled in distribution and I understand that the code is not production ready.

Describe the solution you'd like

  1. Configurable retryable status codes.
  2. Configurable connect timeout, max conns, request timeout, socket timeout, keepalive, etc., to PoolingHttpClientConnectionManager
  3. Connection leak needs to be fixed by making use of closable response
  4. Http compression for outgoing requests.
  5. DLQ as optional setting
  6. Removal of http client building from hot path.
  7. Last missing flush when iterating over records is required.

Describe alternatives you've considered (Optional)
We built custom data prepper distributions by including all the above.

@kkondaka
Copy link
Collaborator

Some basic functionality exists but it needs to be verified and make sure it works in all cases. May also need to add more test cases before this can be included.

@RashmiRam
Copy link
Author

@kkondaka Are there any more items other than the above ones(mentioned in issue description) & tests are planned ?

@kkondaka
Copy link
Collaborator

@RashmiRam I will have to check and get back to you. Do you plan to work on it?

@RashmiRam
Copy link
Author

Yes. I have added those mentioned support in my fork. I haven't added any tests there. If nobody is working on it, I can submit a PR with tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Development

No branches or pull requests

2 participants