server spec for ray llm serving

[Fizzbb]

Hi there,

I am wondering what hardware does ray use for serving in this llmperf leaderboard. Is it cpu or gpu? if it is GPU what's the model?

Thanks,
Fizzbb

[xieus]

The dashboard uses llmperf benchmarking tool to run tests over different LLM APIs.

As mentioned in the llmperf benchmarking tool page, the backend of different API providers might vary widely, this is not a reflection on how the software runs on a particular hardware (could be GPUs, CPUs or custom ASICs). Hope this helps.