Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a Proxy between Kafka clusters with different amount of brokers #111

Open
akorosi opened this issue Sep 23, 2022 · 0 comments
Open

Comments

@akorosi
Copy link

akorosi commented Sep 23, 2022

Hey,
Currently we have a kafka-cluster with 4 brokers in AWS, production environment, that doesn't need any authentication. The consumers and producers are deployed into other aws accounts then the Kafka cluster. The new Kafka cluster is deployed under a different account with 6 brokers and it require SASL authentication. So the clients are running in Aws Account A, old kafka cluster is in Aws Account B, new cluster is in Aws Account C.
Each of the current brokers are listening on port 9093, but have different hostname. Under Aws Account B there is a VPC Endpoint Service that point to the Network Load Balancer of the specific broker. The new Kafka cluster's brokers are listenning on different port, and has the same hostname.
What is your recommendation for the configuration for this situation?

What I tried: Created 4 ECS Services and kafka-proxy is configured in this way:

Service1: Listening on 9093 port and forward traffic to new cluster's broker to 9093 port
Service2: Listening on 9093 port and forward traffic to new cluster's broker to 9094 port
Service3: Listening on 9093 port and forward traffic to new cluster's broker to 9095 port
Service4: Listening on 9093 port and forward traffic to new cluster's broker to 9096 port

it means 2 brokers of the new cluster are not configured in kafka-proxy.
If I configure the kafka-proxy to use the cluster's bootstrap broker that is listening on port 9092, I have to define all of the brokers too in the proxy (without it the service fails, as the broker lists are received from the kafka metadata and the consumer's cannot connect to)

VPC Endpoint Services are rerouted the new Services' NLB. It seems this solution is working properly, but I'm not sure if it's the most optimal one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant