100B+ Models on 12GB GPU: Strata, Maya & llama.cpp MoE

What if I told you that a graphics card with just 12 gigabytes of VRAM could run an AI model with more than 100 billion parameters? And not just run it, but generate text at speeds approaching 100 tokens per second? Until recently, running models this large locally required expensive professional GPUs, multiple graphics cards, or machines with enormous amounts of memory. But that's starting to change. New inference engines like Strata, Project Maya, and recent improvements to llama.cpp are making it possible to run massive AI models on surprisingly modest hardware. And the secret isn't necessarily faster GPUs. It's a smarter way of managing memory and computation. Running large language models locally has several advantages. You get more control over your data, you don't need to pay for every token through an API, and you can run models without depending on an internet connection. The challenge has always been hardware. A typical consumer graphics card might have 12 or 16 gigabytes of VRAM. That's plenty for smaller language models, but nowhere near enough for models containing hundreds of billions of parameters. To understand why, consider a 100-billion-parameter model. Even at 4-bit quantization, its weights alone require approximately 50 gigabytes of memory. And that's before accounting for the additional memory needed during inference. Traditionally, the solution was to buy a GPU with more VRAM, distribute the model across multiple GPUs, or offload parts of the model into system RAM. But offloading often comes with a significant performance penalty. This is where Mixture-of-Experts models become interesting. Unlike traditional dense models, which activate essentially all their parameters for every token, Mixture-of-Experts models divide much of their computation among specialized groups of parameters called experts. For each token, the model selects only a small subset of these experts. Take Qwen3.8-Flash-Next, for example. It contains approximately 125 billion parameters, but only around 6 billion are active for each token. That means the model doesn't need to perform calculations using all 125 billion parameters every time it generates a token. However, there's still a problem. Even though only a fraction of the experts are active, their weights must remain accessible. And the model can select different experts for every token. If those weights are constantly moving between system RAM and GPU memory, the transfers can become a major bottleneck. This is exactly the problem Strata was designed to solve. Strata is an open-source inference engine specifically optimized for Qwen3.8-Flash-Next. Instead of requiring the entire model to reside in GPU memory, Strata distributes the workload across GPU VRAM, system RAM, and the CPU. Frequently used experts can remain cached in GPU memory, where they can execute extremely quickly. Other experts remain in system RAM, where the CPU can execute them without necessarily transferring their weights to the GPU first. This is an important distinction. Traditional GPU offloading often involves moving weights into GPU memory before performing calculations. Strata can instead execute some experts directly on the CPU while the GPU handles other work. By combining intelligent expert caching with heterogeneous CPU and GPU execution, Strata reduces unnecessary data movement and makes better use of the hardware already available. It also incorporates speculative decoding techniques, allowing the system to predict and verify multiple tokens to accelerate generation. The result is impressive. On an RTX 5070 with just 12 gigabytes of VRAM and 64 gigabytes of system RAM, Strata has reported generation speeds of approximately 94 tokens per second using a 2-bit quantization of Qwen3.8-Flash-Next. That's a model with more than 100 billion total parameters, running on a mainstream consumer GPU. And this helps explain why Strata has attracted so much attention in the local AI community. It challenges the assumption that running a massive language model necessarily requires massive amounts of VRAM. But Strata isn't the only project exploring this approach. Project Maya takes the underlying ideas further. Built on Strata, Project Maya targets GLM-5.3-Flash, a much larger Mixture-of-Experts model containing approximately 321 billion parameters. Because the model is substantially larger, Project Maya introduces a more aggressive memory hierarchy. Frequently accessed experts can reside in GPU VRAM. Additional experts are cached in system RAM. And experts that don't fit into either memory pool can be retrieved from fast NVMe storage. This allows Project Maya to work with systems that have considerably less physical memory than the model's total size. Its documented minimum system RAM requirement is 32 gigabytes, although performance depends heavily on available VRAM, RAM, and SSD speed. Project Maya is particularly interesting because it demonstrates that even models approaching a third of a trillion parameters can potentially run on consumer hardware. However, running a model and running it quickly are two very different things. The more frequently experts must be retrieved from SSD storage, the greater the potential performance penalty. Meanwhile, another important development is happening inside llama.cpp. In October 2026, llama.cpp integrated a new GPU expert-caching mechanism based on a least-recently-used cache, commonly called LRU. The idea is straightforward. Frequently accessed experts remain in GPU memory. When the model needs an expert that isn't cached, its weights can be transferred from system RAM into VRAM. If the cache is full, the least recently used experts are evicted to make room. Unlike Strata and Project Maya, which are specialized for particular model architectures, llama.cpp supports a broad range of language models. That makes its new expert cache especially interesting. The same general caching mechanism can potentially benefit many supported Mixture-of-Experts architectures, rather than just one model family. Although llama.cpp's implementation is not a direct derivative of Strata, it addresses the same fundamental challenge: how to use limited GPU memory efficiently when running very large MoE models. So how do these approaches compare in practice? Let's focus specifically on graphics cards with 12 to 16 gigabytes of VRAM, paired with no more than 64 gigabytes of system RAM. Starting with Strata, an RTX 5070 with 12 gigabytes of VRAM and 64 gigabytes of RAM has reported approximately 94 tokens per second using Q2_0 quantization. With IQ2_XS quantization, reported performance is around 79 tokens per second. And moving to the higher-quality IQ3_S quantization, performance drops to approximately 53 tokens per second. These are impressive results, especially considering the size of the model. On the llama.cpp side, an RTX 5060 Ti with 16 gigabytes of VRAM and 62 gigabytes of system RAM produced approximately 15 tokens per second without the new MoE cache. Enabling an 8-gigabyte expert cache increased generation speed to approximately 19 tokens per second. That's an improvement of roughly 29 percent. Strata's reported generation speeds are substantially higher, although these aren't controlled, apples-to-apples benchmarks. The GPUs, quantizations, and inference optimizations differ, and Strata's speculative decoding can provide an additional advantage. For Project Maya, comparable single-GPU benchmarks within these exact hardware limits are still difficult to find. While its architecture supports low-memory configurations, there isn't enough verified performance data to make a meaningful speed comparison against Strata or llama.cpp on 12- and 16-gigabyte GPUs. And that brings us to another important question. Which quantization should you actually use? Quantization reduces the precision of model weights, allowing them to occupy less memory. Lower-bit quantizations generally require less memory and can improve performance, although they may also reduce model quality. For Strata, systems with 32 gigabytes of RAM should start by investigating the reduced-expert Coder variant, using its low-memory configuration. This applies whether the GPU has 12 or 16 gigabytes of VRAM. With 64 gigabytes of system RAM, the options become considerably more interesting. IQ2_XS offers a good balance between memory consumption, quality, and speed. For users prioritizing model quality, IQ3_S is another option, although it requires more memory and may generate tokens more slowly. For Project Maya, the most practical starting point across these hardware combinations is Maya-S24. This variant is specifically designed for GPUs with 24 gigabytes of VRAM or less. However, its model files occupy approximately 95 gigabytes. That means even a system with 16 gigabytes of VRAM and 64 gigabytes of RAM cannot hold the entire model in physical memory. Project Maya must rely on SSD-backed expert storage, and actual generation speeds will depend on how frequently those experts need to be retrieved. So yes, 16 gigabytes of VRAM and 64 gigabytes of system RAM can potentially run Maya, but it shouldn't be confused with a configuration that guarantees fast inference. For llama.cpp, quantization recommendations depend on the specific model. If you're running Qwen3.8-Flash-Next, lower-bit GGUF quantizations are the most realistic starting point on these systems. With 64 gigabytes of RAM, IQ2 and IQ3-class quantizations become worth investigating. With only 32 gigabytes of RAM, even low-bit versions of very large models may depend heavily on memory-mapped storage. The new GPU expert cache can improve performance, but it doesn't eliminate the underlying memory requirements. One advantage of llama.cpp is that you aren't limited to enormous models like Flash-Next. You can also choose smaller Mixture-of-Experts models that fit much more comfortably within your available memory. And beyond these three projects, several other inference engines are exploring related techniques. KTransformers specializes in heterogeneous CPU and GPU inference, using optimized CPU kernels to execute MoE experts efficiently. Siphon.cpp experiments with streaming expert weights between storage, system memory, and GPU memory. TinyGiant explores SSD-backed inference for models that exceed available physical memory. The llama-moe-cache project investigates expert caching and prefetching within the llama.cpp ecosystem. And Strata-4GPU extends Strata's approach toward systems with multiple graphics cards. These projects vary considerably in maturity and performance, but they all reflect a broader shift in local AI inference. Instead of assuming that every parameter must remain in expensive GPU memory, developers are finding smarter ways to distribute weights and computation across the entire system. And that changes how we should think about hardware requirements. A 12-gigabyte GPU is no longer automatically a barrier to running a model with more than 100 billion parameters. With the right inference engine, sufficient system RAM, and an appropriate quantization, models of this size can deliver surprisingly useful performance. But VRAM is only one part of the equation. System RAM capacity, CPU performance, memory bandwidth, SSD speed, quantization, and inference-engine optimizations all contribute to the final result. And while Strata currently demonstrates particularly impressive performance for Qwen3.8-Flash-Next, Project Maya is pushing the boundaries of model size, while llama.cpp is bringing similar caching techniques to a much broader collection of models. The bigger story is that local AI is becoming less dependent on having the largest possible graphics card. The future may not be about fitting an entire model into VRAM. It may be about intelligently deciding which parts of the model need to be there, which parts can run elsewhere, and how to keep everything working together efficiently. And that's why a GPU with just 12 gigabytes of VRAM can now accomplish things that once seemed completely out of reach.