News

By Uwe David, Platform Architect, Cyberlink Ltd

Under load – models, engines and multi-GPU inference on RTX PRO 6000 Blackwell

Local AI sounds like a hardware question at first: enough fast GPUs, enough VRAM, install a large language model and off you go. On paper an RTX PRO 6000 Blackwell Max-Q with roughly 96 GB of VRAM already looks quite comfortable for that. With several of these cards the matter seems settled for good.

In practice, that is exactly where the interesting work begins.

At Cyberlink we tested seven current language models with three inference engines on NVIDIA RTX PRO 6000 Blackwell Max-Q. Through our partnership with cloudscale we had two GPU flavors available for this, with two and four identical GPUs respectively. The measurement series cover single-GPU inference, multi-GPU operation, long contexts and load tests with up to 256 concurrent requests.

What interested us was less another synthetic GPU benchmark than the question of what actually works with current hardware and current models, and where the limits lie in practical operation.

One finding ran through almost every test: with local AI the GPU is only one part of the system. Model architecture, memory footprint and inference software determine at least as strongly what ends up working.

On a single GPU, three engines running the same model reached almost the same speed for a single user. Under 32 concurrent requests, the gap between the fastest and the slowest configuration suddenly came to almost a factor of six. A model with 90.7 GiB of weights nominally fitted into the memory of one card and still could not be operated. Two GPUs either ran at full load or, depending on the software, stayed at around 45 percent utilization regardless of the number of requests. Simply switching off thinking raised the number of complete answers per second by a factor of 3.5 in one measurement series.

The hardware sets the frame. How much of it becomes usable is decided by the software stack.

The test environment

Testing was done on two cloudscale GPU flavors:

	GPU2-256-40-2-1600	GPU2-512-80-4-800
GPUs	2 × RTX PRO 6000 Blackwell Max-Q	4 × RTX PRO 6000 Blackwell Max-Q
VRAM per GPU	97,887 MiB	97,887 MiB
vCPU	40	80
System RAM	251 GB	512 GB
GPU interconnect	PCIe Gen5 ×16	PCIe Gen5 ×16
Storage	1.6 TB scratch + 193 GB system	800 GB scratch + 200 GB system

Both flavors run virtualized, which means the GPUs communicate over PCIe. On paper this is a comfortable starting position for local inference: plenty of VRAM, current Blackwell GPUs and enough CPU and main memory for large models. The tests quickly show, however, why the capability of such a system cannot be derived from GPU count and VRAM alone.

The inference software used was vLLM, SGLang and llama.cpp. Driver versions, builds, quantizations, startup parameters and the complete measurement series are documented in the repository. For context, the effects are initially more interesting than the individual command-line parameters.

When a 90 GB model does not fit on a 96 GB GPU

VRAM is the most obvious limit in local inference. But the equation "model smaller than VRAM = model runs" does not hold.

GLM-5.3-Flash provided a good example: an NVFP4 variant we tested occupied around 90.7 GiB per card. The RTX PRO 6000 provides just under 96 GiB. The weights could be loaded, yet the remaining memory was not enough for operation. Beyond the model weights themselves, the inference engine needs further memory. An important part of that is the KV cache, which holds information from the context already processed. A different quantization of the same model occupied around 73 GiB and ran.

The same underlying problem appeared with long contexts. Gemma-4-31B ran on one card with 32,768 tokens of context. On doubling to 65,536 tokens the KV cache alone grew to around 38 GB and the configuration no longer fitted.

For capacity planning, at least three quantities are therefore relevant: model weights, runtime overhead and the desired context.

Quantization is more than compression

Models do not have to be operated at the original numerical precision of their weights. FP8, NVFP4 or the various GGUF formats reduce the memory footprint, in some cases considerably. That sounds at first like a classic trade-off between memory footprint and quality. For local inference, however, quantization has a second meaning: it can decide whether a particular hardware configuration is possible at all.

In our measurements the memory footprint of the models tested ranged from under 20 GiB to well over 100 GiB per GPU. Qwen3.5-397B-A17B came to around 117 GiB per card even as NVFP4. That settled the matter before the first benchmark could start. Other models with more than 100 billion parameters, by contrast, ran without difficulty.

The parameter count alone is therefore only of limited use for estimating the hardware required. There is a second reason for that.

116 billion parameters need not mean 116 billion calculations

Many current large models use a mixture-of-experts architecture, MoE for short. A large number of specialized sub-networks, so-called experts, exist, of which only a few are activated per token. MiniMax-M2.5, for instance, has 116 billion parameters and 256 experts. Eight of those experts are used per token. The model's weights still have to be stored somewhere, but only part of the whole model becomes active for the actual computation.

A direct comparison with Gemma showed how large the effect can be. We ran a sparse Gemma model and a dense variant of a similar size on the same GPU and with the same engine. The sparse model activates around 4 billion parameters per token; the dense model computes its full parameter set. At 32 concurrent requests the sparse model reached 5.45 complete answers per second, the dense variant 0.80 answers per second. That is almost a factor of seven on the same hardware.

Model size alone therefore says less and less about the actual compute cost.

Same GPU, same model, almost a factor of six

One of the most revealing tests required no change of hardware at all. We used the same RTX PRO 6000, the same model and the same load. The only thing changed was the inference engine. Gemma-4-26B-A4B ran with vLLM, SGLang and llama.cpp.

With a single request the three engines were still relatively close together:

Engine	Answers/s	GPU utilization
vLLM	3.08	98 %
llama.cpp	2.98	73 %
SGLang	2.67	93 %

For a single user all three variants would be fast. At 32 concurrent requests the picture looks considerably different:

Engine	Answers/s	GPU utilization
vLLM	31.88	100 %
SGLang	21.55	76 %
llama.cpp	5.45	38 %

What was a relatively small difference with one user became a difference of almost a factor of six under load. It also shows in the latency. The median for a complete answer at this load was 1.03 seconds with vLLM, 1.56 seconds with SGLang and 6.48 seconds with llama.cpp.

For practical operation this behavior is precisely what matters. A developer trying the system out alone might conclude, for all three engines: fast enough. With 30 or 50 simultaneous users the situation looks different. The engine decides not only how quickly a single request is processed, but also how well many requests are batched together and how busy the available GPU is kept.

More GPUs mainly mean more options, not automatically more performance

A large model can be distributed across several GPUs, but additional cards do not automatically double throughput. How well this works depends strongly on model architecture and inference engine.

This showed particularly clearly with llama.cpp: Qwen3.8-Flash-Next did distribute across two RTX PRO 6000, but both GPUs stayed at a constant 45 percent utilization even as load increased. More concurrent requests changed virtually nothing about that. MiniMax-M2.5 under vLLM demonstrated the counter-example on the same hardware and kept both GPUs fully occupied.

Additional GPUs therefore mean, first and foremost, additional capacity. Models that do not fit into the memory of one card can be distributed, and considerably larger context windows become possible. On four RTX PRO 6000, for example, we were able to operate a large DeepSeek model with a context of 1,048,576 tokens and, in a needle-in-a-haystack test, successfully retrieve information at 10, 50 and 90 percent of the context. The full million-token prefill took around 471 seconds.

That is precisely where the more important insight from the multi-GPU tests lies for us: more GPUs first of all widen the solution space. Whether correspondingly more speed comes out of it is decided by the software stack.

"Thinking" is also a capacity question

With reasoning models another factor comes into play. A model can internally generate considerably more tokens than later become visible as the actual answer. This additional work can be worthwhile for complex tasks, but it costs compute time.

On the four-GPU configuration we measured the same workload with different thinking settings. With thinking disabled, 3.5 times more complete answers per second were processed. The main reason for this was that 8.75 times fewer tokens were generated per answer.

For a production AI service this is relevant. Not every task needs the same amount of reasoning. A complex analysis, debugging task or planning exercise may benefit from it; a simple information lookup may not. Thinking is therefore not only a property of the model, but also a question of available capacity and desired response time.

The software moves almost as fast as the models

Another part of the tests consisted of things that did not run in the first place. That sounds unspectacular, but with current AI infrastructure it is an important finding.

One model fitted into memory but was not yet supported by the engine. With another, the model architecture was known in principle, but a required GPU kernel for Blackwell was missing. A further configuration started without trouble and only failed under higher load.

Blackwell is current hardware. At the same time, new model architectures sometimes appear faster than the various inference engines can fully support them. "The model fits on the GPU" therefore does not by any means yet mean "the model runs with this engine on this GPU".

Local inference has long been a multi-user topic

DeepSeek-V4-Flash-0731 reached up to 32.58 complete answers per second under load on two RTX PRO 6000, at 256 concurrent requests. The median latency was just under eight seconds. High concurrency therefore increases overall throughput, not automatically the speed for the individual user. The measurement does show, however, that local inference has long gone beyond the single developer with a GPU workstation and can be operated as a genuine multi-user service.

That also shifts the planning question. Instead of only asking "which GPU do we need?", it becomes a matter of which combination of model, quantization, context, reasoning, engine and hardware fits the actual workload.

Conclusion

On paper the RTX PRO 6000 Blackwell Max-Q offers a comfortable basis for local AI. Our tests confirm that very large models, high concurrency and context windows reaching into the range of one million tokens are technically possible with it.

The more interesting insight, however, lies one level above: the GPU sets the upper limit. The software stack decides how close you get to it. Model architecture, quantization, context, inference engine, scheduler and multi-GPU strategy can make the practical difference between "does not fit", "runs" and "runs really well".

That is why we now regard local AI less as a procurement topic and more as a platform topic. The question is not only which GPU can execute a model. The more interesting question is: which combination of model, engine and hardware delivers the most sensible service for the specific use case?

That is exactly where the real work begins for us.

Technical details and raw data

Anyone who wants to go deeper will find the complete configurations, engine versions, quantizations, benchmark protocols, raw data, failed attempts and hypotheses examined in the repository: github.com/flash-ultra/ai-server-setup.