Benchmarking Suite

[tubarao312]

• The server and each SDK has a benchmark suite.
• This benchmark suite gets ran in the CICD every time a PR is opened to main.
• The results of the CI benchmarks are registered and saved for historical comparisons. See https://github.com/benchmark-action/github-action-benchmark?tab=readme-ov-file

Benchmarks Types

There are two types of benchmarks:

• Server --> Benchmarks just the server. Implemented using cargo bench
• E2E --> Benchmarks from the SDKs. Implementation depends on the language. Because every client should have the same behavior, there should exist a general blueprint so we can compare each language's benchmark as well as compare to previous results so we can monitor performance improvements and downgrades.

Server Benchmarks

• Task creation speed
• Task query speed

E2E SDK Benchmarks

Publisher

• Task creation speed
• Task result query speed

Worker

• Task pulling speed
• Task result submission speed

E2E

• Time to complete 10k tasks with N workers (we can use a matrix method to test with 1, 2, 4, 8, 16, 32 workers).

[linear]

TAQ-36 Create benchmarks