Kinesis Firehose Processor

[danrubenstein]

Hi @flashmob!

This is a great project, really looking forward to working with it.

I'm wondering if you would be willing to consider a PR that would add a processor for Kinesis Firehose (or perhaps Kinesis Streams). One thing that I'd like to be able to do is put messages into a log to examine them at a later time (and perhaps by arbitrary consumers), and Firehose / Streams seems like a good way to help build that bridge.

Let me know if this is something you'd consider, and I'll start kicking around some draft code.

Thanks!

[flashmob]

Hi, There is a PR that will add the ability for streaming using the io.Writer interface. #135 Would be great to have someone try it out and see how it works in practice, get some feedback and review. Willing to assist answer any questions in return. Once you have your own processor, there is no need to merge it upstream. You can develop the processor in your own project, the use the go-guerrilla package as an API, it includes method for adding the processor and enabling it via configuration.

…

On Thu., 7 Mar. 2019, 10:39 Dan Rubenstein, ***@***.***> wrote: Hi @flashmob <https://github.com/flashmob>! This is a great project, really looking forward to working with it. I'm wondering if you would be willing to consider a PR that would add a processor for Kinesis Firehose (or perhaps Kinesis Streams). One thing that I'd like to be able to do is put messages into a log to examine them at a later time (and perhaps by arbitrary consumers), and Firehose / Streams seems like a good way to help build that bridge. Let me know if this is something you'd consider, and I'll start kicking around some draft code. Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#138>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAnmP54tWJ1XJQ6-sGJqFlAcYe9Z4mrBks5vUFG4gaJpZM4biC26> .

[danrubenstein]

Sounds good - I'm sure it'll take me a bit to grok this, but thanks for pointing me in the right direction. I'm going to go ahead and close this for now, and pick up any specific questions I have in new issues.

Thanks!

[flashmob]

All right. We can communicate on the comments of the PR. You can check out the branch with the PR and work from there. There's also an example added to the bottom of api_test.go From the sounds of it, all you need to do is have a Firehose io.Writer and connect that up in similar way as the examples.

…

On Thu., 7 Mar. 2019, 13:33 Dan Rubenstein, ***@***.***> wrote: Closed #138 <#138>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#138 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAnmPzQlYZVwgjuFqxmPG90-T4aBhs4Qks5vUHp6gaJpZM4biC26> .

[danrubenstein]

Upon further reflection here - the processor model is sufficient for what I'm looking for - I may come back to the writer later on, but it's not super pressing for me.

Thanks!