Eventing Trigger - dispatcher rate limiter

[lamluongbinh]

Hi Team,

I am currently working on applying a rate limiter to control the rate of messages process by a Trigger.

Based on my tests, this can be achieved by:

1. Enabling the feature flag: dispatcher-rate-limiter: enabled.
2. Creating a new broker with 1 partition.
3. Adding an annotation to the Trigger to ensure it is ordered: kafka.eventing.knative.dev/delivery.order: ordered.
4. The limit is determined by the following equation: LimitedNumber = NumberOfPartitions * 20

The above finding is incorrect. I will update it below.

I would like to ask if there is another way to control the LimitedNumber.

From my reading of the code, the bucket is calculated a: link

bucket = vReplica * (MaxPollRecords or NumberOfPartitions)

Alternatively, is there a way to apply a configuration or annotation to set the limit number for a Trigger?

[pierDipi]

Hi @lamluongbinh

I would like to ask if there is another way to control the LimitedNumber.

From my reading of the code, the bucket is calculated a: link

bucket = vReplica * (MaxPollRecords or NumberOfPartitions)

Alternatively, is there a way to apply a configuration or annotation to set the limit number for a Trigger?

currently, there is not, however, I'm wondering if the way is configurable is working for you

Based on my tests, this can be achieved by:

1. Enabling the feature flag: dispatcher-rate-limiter: enabled.
2. Creating a new broker with 1 partition.
3. Adding an annotation to the Trigger to ensure it is ordered: kafka.eventing.knative.dev/delivery.order: ordered.
4. The limit is determined by the following equation: LimitedNumber = NumberOfPartitions * 20

from where did you get that 20 ? I think you should see 1 event per second in this case

[lamluongbinh]

Hi @pierDipi,

Thanks for your reply.

The number is showing in my test, but I'm not sure where the value of 20 comes from. Could it be the default worker pool size for Vert.x (20 threads)? Initially, I tested with 1 partition, and then increased it to 3 partitions. The behavior is as follows:

1. With 1 partition: The rate limit is set to 20 messages per second.
2. After increasing to 3 partitions (without sending new messages): No messages are allocated to partitions 2 and 3. As a result, the rate limit remains at 20 messages per second.
3. When new messages are sent: They start filling partitions 2 and 3, causing the rate limit to increase to 60 messages per second as more partitions contribute to processing.
4. Once partitions 2 and 3 are processed (and no new messages arrive): The rate limit drops back to 20 messages per second, aligning with the load on the active partitions.

The above finding is incorrect. I will update it below.

[pierDipi]

which version are you using?

[lamluongbinh]

Hi @pierDipi, sorry, I forgot to add my version:

• knative-operator: v1.13.3
• knative-eventing: 1.13.7
• eventing-kafka-broker: v1.13.11

[lamluongbinh]

Hi Team,

Could we implement a feature to control the rate limit for Knative triggers and also provide clear documentation to guide users on how to configure and manage their rate limits effectively?

Why is this needed?
There are scenarios where an excessive number of messages might be sent to a service endpoint simultaneously, causing high pressure on the service. This can lead to cascading effects, such as:

• Overloading the service itself, leading to failures or degraded performance.
• Increasing stress on downstream systems like databases, Redis, or other dependencies.
• A domino effect that propagates failures across the system.

Having a configurable rate limit at the trigger level would help mitigate these risks by throttling the flow of messages and ensuring the service operates within manageable limits.

Looking forward to hearing your thoughts on this.

[lamluongbinh]

Hi @pierDipi,

Previously, I mentioned that the load test showed a rate of 20 messages per second for each partition. However, this result turned out to be completely incorrect.

During further testing, I discovered that the rate was caused by a limitation set by max.partition.fetch.bytes=65536. With this fetch size, my message size of around 8KB–10KB (as I recall), and the consumer logic fetching messages multiple times (every 200ms), this coincidentally resulted in the round number 20 being displayed on Datadog.

Moreover, the rate limiter is not applied to Knative triggers or the Knative dispatcher because we currently lack the control plane logic to enable the flag and the data plane logic to enforce the correct rate. To address this, I added logic to control the Knative trigger rate limit and applied it to my system.

• My patch code to enable the rate limiter and precisely control the rate can be found here: https://github.com/knative-extensions/eventing-kafka-broker/pull/4208/files

This is my first time contributing and adding code logic to a public repository. If you notice any mistakes in my code, please feel free to let me know.

Thanks to the team for building this awesome project!