Weights are only the first memory cost
Pin the model revision and actual checkpoint format, then budget for all resident weights. For a mixture-of-experts model, active parameters per token are not the full checkpoint size. Inference also needs cache, model state, workspace and prefill headroom. Weight quantization and KV-cache precision are separate choices, subject to artifact, engine and kernel support.
Context and concurrency both consume cache
The context budget includes input and output tokens. Concurrent generation requests are not employee headcount. Conventional full-attention cache generally grows with occupied context and active requests; hybrid, sparse and compressed attention need architecture-specific accounting. A model context limit does not guarantee that arbitrary hardware can serve many requests at that length.
Multiple GPUs do not form an unrestricted memory pool
Tensor parallelism distributes one model across GPUs; separate serving replicas each retain their own weights. Some GQA models replicate KV heads when the tensor-parallel degree exceeds the KV-head count, so cache does not keep shrinking in proportion to GPU count. Check the busiest GPU as well as system totals and validate the partition rules.
Validate with your own requests before purchase
Fix the GPU variant, model, precision, engine, input/output lengths and concurrency. Measure peak memory, time to first token, sustained decoding and errors. Validate extended context with the official runtime recipe. A configurator estimate can rule out clearly unsuitable combinations, but cannot replace testing or a contractual acceptance result.
Frequently asked questions
Select a question to expand its answer.
Why can several high-memory GPUs still run out of memory?
Check GPUs per replica, cache replication, context length, concurrency and per-device headroom. Aggregate VRAM alone is insufficient.
Does INT4 reduce all memory use to one quarter?
No. Cache and runtime state do not automatically shrink with weight precision, and quantized artifacts may include unquantized tensors and auxiliary data.
Sources & scope
Guides do not replace the exact model documentation, verified bill of materials or a written quotation.