cloudscaleKnowledge Base
Model Performance on Our GPUs – How Many Tokens/s Can Be Expected
There is no single number – token speed depends on the model, its size, and how you run it.
Honest answer: It depends on a lot of things, and we mean a lot: which model, its architecture and family, its size, the quantization (fp8, int4, ...), how long your prompts and outputs are, how many requests run at once (batching), the context window, the inference software, KV cache settings, and how many of our GPUs you spread the model across. Change any one of those and the number moves, sometimes quite dramatically.
So instead of quoting a single figure that would be wrong for most people, we point you at a free third-party estimator that lets you turn all those knobs yourself. It is an independent community tool (we do not run it and are not affiliated with it), but it is a good sanity-check.
Our offering is 1-4× NVIDIA RTX PRO 6000 Max-Q (96 GB VRAM each), connected over PCIe. We have pre-filled two examples on that hardware so can explore some examples:
- A Gemma model on a single card: an MoE design that stays fast and responsive because only a small part of it runs per token. Uses about a third of one card's memory here, leaving room to spare.
- A large Qwen model across four cards: more capable, but each response comes several times slower.
Open either link, tweak the settings to match your workload, and you will get a grounded estimate.
Important: These are estimates only; real throughput depends on your exact workload and software stack. Please benchmark directly on our hardware.