Local LLM VRAM Budgets: 8, 12, 16, 24 & 32 GB GPUs

How do you get an open-weight AI model running locally on your GPU? And how do you figure out what will actually fit? The biggest constraint is usually VRAM. And for maximum performance, ideally you want the important parts of inference to stay on the GPU rather than spilling into much slower system RAM. There are three major things competing for that VRAM. The model weights, the KV cache, and runtime overhead. In other words, model weights plus KV cache plus runtime overhead needs to fit inside your available VRAM. Let's start with the model. If you're using llama.cpp, LM Studio, or similar software, you'll frequently encounter models in the GGUF format. GGUF is the file format, but the weights inside that file can be stored at different levels of quantization. You'll see names like Q8, Q6, Q5, Q4, Q3, and more specialized variants like IQ3. Higher precision generally means better preservation of the original model, but also larger weights. Lower precision means a smaller model, but eventually you start sacrificing quality. But the weights aren't the whole story. The second major component is the KV cache. The model weights are basically fixed after you load them. The KV cache is working memory used as the model processes your conversation or prompt. And its size depends heavily on your context window. If you configure 32,000 tokens of context, you need less KV memory than at 64K, 128K, or 256K. So don't allocate a massive context window just because the model supports one. If you only need 32K, allocating 128K can consume VRAM that could have gone toward higher-quality model weights. The KV cache can also have its own quantization. So you might have Q5 model weights with a quantized KV cache, for example. That gives us two independent tradeoffs: how precisely we store the model itself, and how much memory we're willing to spend on context. Finally, there's runtime overhead. llama.cpp, CUDA, Flash Attention, compute buffers, and other allocations need additional GPU memory. So a 15-gigabyte GGUF doesn't necessarily fit comfortably on a 16-gigabyte GPU. You need some breathing room. And there's another distinction worth making. llama.cpp can offload parts of a model or the KV cache into system RAM. So when somebody says they're running a huge model on an 8-gigabyte GPU, that can absolutely be true. But "it runs," and "everything fits in VRAM" are not the same thing. For this example, our goal is maximum GPU resident performance. So let's make this concrete using Qwen 3.8. These are my personal balanced starting points. They're not rules. You can trade model quantization, KV precision, and context length depending on your workload. On an 8-gigabyte GPU, I wouldn't try to force the 27B model onto the card. I'd use Qwen 3.8 9B at around Q4 with roughly a 32K context window and a quantized KV cache. The weights are around six gigabytes, leaving the rest for KV cache, runtime, and some safety margin. At 12 gigabytes, I'd still use the 9B model, but now I'd increase the weight quality. Something around Q6 with 64K context is a much more balanced setup than aggressively crushing the 27B model just to say that it fits. At 16 gigabytes, though, I'd make a different tradeoff. Now I'd move up to Qwen 3.8 27B. An IQ3 class version can put the model weights at roughly 13 to 14 gigabytes. That's aggressive quantization, but my priority here is getting the intelligence of the larger 27B model onto a 16 gigabyte GPU. I'd start with around 32K context and a quantized KV cache. You might be able to push that higher depending on your exact quantization and runtime configuration, but 32K is my conservative starting point. At 24 gigabytes, the 27B model becomes much more comfortable. I'd use approximately Q5 weights, which come in around 21 gigabytes, combined with around 64K context and quantized KV. But here's where these tradeoffs become interesting. If context matters more to you, you could drop from Q5 to Q4 and potentially spend that saved VRAM on something closer to 128K context. Neither configuration is universally better. One prioritizes weight precision, the other prioritizes working context, and finally on a 32-gigabyte GPU, you might think I'd simply choose Q8, but I probably wouldn't. Q8 weights for this 27B model are already around 29 gigabytes. Yes, the model weights technically fit, but we've left almost nothing for context and runtime. Instead, I'd choose approximately Q6 at around 23 to 24 gigabytes and use the remaining memory for something like a 128K context window, quantized KV cache, and comfortable runtime headroom. And that's the key lesson. Don't simply ask, what's the highest quality GGUF that fits on my GPU? Think of VRAM as a budget. First, choose the model intelligence you need, then choose an appropriate weight quantization, then decide how much context your workload actually requires, and how much memory you're willing to spend on the KV cache. And finally, leave enough space for the inference runtime itself. For my personal rubric, that means roughly 8 gigabytes, 9B Q4 32K, 12 gigabytes, 9B Q6 64K, 16 gigabytes, 27B IQ3 32K, 24 gigabytes, 27B Q5 64K, and 32 gigabytes, 27B Q6 around 128K. But these aren't magic configurations. If you're doing simple chat, maybe you only need 16K or 32K context and can spend more VRAM on weight quality. If you're running a coding agent or analyzing huge documents, perhaps you sacrifice some weight precision to get 128K or even more context. And everything we've discussed here is specifically for Qwen 3.8. Different models have different architectures, parameter counts, KV cache requirements, and quantization options. But the method stays the same. Model weights, plus the KV cache for your chosen context, plus runtime overhead, needs to fit inside your VRAM. Once you understand those three pieces, figuring out what model your GPU can actually handle becomes much easier.