Qwen3.8-Flash-Next changes the local inference equation: a huge model with only a small fraction of its parameters active for each token. This video explores its mixture-of-experts design, Gated DeltaNet and attention, Qwen Sparse Attention, HyperConnections, predictive lookup embeddings, and multi-token prediction—and why supporting the architecture is only the beginning.
We compare the approaches of llama.cpp, SGLang, EXL3, Trellis and Strata, from broadly portable compatibility to specialized execution and intelligent use of GPU memory, CPU compute, system RAM and SSD storage.
The central question: if only a small portion of an enormous model is needed at any moment, what is the smartest way to use the hardware we already have?
00:00 A different inference problem 00:13 Total parameters versus active parameters 00:32 Gated DeltaNet, attention and the hybrid architecture 00:55 PLE: a massive lookup system 01:18 MTP: propose, verify, accept 01:40 Why inference support is difficult 02:08 llama.cpp and the Qwen4Exp milestone 02:32 SGLang and day-zero support 02:52 Compatibility versus optimization 03:26 EXL3 and Trellis: specialized execution 04:03 Strata: GPU, CPU, RAM and SSD 04:33 Concurrent CPU and GPU computation 04:57 Combining PLE and multi-token prediction 05:19 Comparing the approaches 06:03 How specialized ideas reach mainstream engines 06:14 Why Flash-Next changes the hardware equation 06:45 Making smarter use of the hardware we have
AI Generated·May contain errors
Video Summary
AI Generated
QEN 3.8 FlashNext is a large Mixture of Experts (MoE) model with over 100 billion parameters, designed for efficiency by activating only 6 billion parameters per token. It introduces novel architectural elements like PLE, QSA, and multi-token prediction, challenging traditional inference engines to optimize hardware utilization across GPUs, CPUs, and storage.
Qwen3.8-Flash-Next is interesting not just because of its performance, but because its
architecture is significantly different from the models that local inference engines were
originally designed to run. Flash-Next is a large mixture-of-experts model. It has well over a hundred
billion total parameters, but only around 6 billion are active for each token. That means the entire
model has to exist somewhere, but only a relatively small portion of it actually needs to perform
computation at any particular moment. But mixture-of-experts is only one part of the story.
Flash-Next introduces several architectural ideas associated with the direction
Qwen is taking toward Qwen4. It combines Gated DeltaNet with conventional attention layers,
and adds Qwen Sparse Attention, or QSA, HyperConnections, and a very large predictive lookup
embedding system known as PLE. PLE is particularly unusual. It contains tens of billions of
parameters, but behaves more like a gigantic lookup structure than a conventional neural-network
layer. Because only small portions of it are required at a time, an inference engine can
potentially keep it in system memory or even on fast storage rather than permanently consuming
GPU memory. Flash-Next also supports multi-token prediction, or MTP. Instead of generating exactly
one token at a time, a smaller prediction component can propose several future tokens.
The main model verifies those predictions, and when they are correct, multiple tokens can effectively
be accepted together. That can significantly increase generation speed. All of this creates
challenges for existing inference engines. Supporting Flash-Next isn't simply a matter of adding
another model name. The runtime has to understand its hybrid attention architecture, HyperConnections,
QSA, PLE, mixture-of-experts routing, recurrent state, and eventually its multi-token prediction
system. The good news is that mainstream inference engines are already adding this support.
For llama.cpp, the major milestone came on August 27th, 2026, when the core
Qwen3.8-Flash-Next architecture was merged upstream under the internal name Qwen4Exp.
That implementation added the model conversion and execution support necessary for features
including HyperConnections, QSA and PLE. SGLang moved at roughly the same time. It had a
day-zero Flash-Next implementation and dedicated Docker image around August 26th,
immediately following the model's release. That initial support continued evolving through
additional pull requests and was subsequently incorporated more formally into SGLang's
model-support stack. But there's an important distinction here. Supporting a model correctly is not the
same thing as running it optimally. llama.cpp and SGLang are large, general-purpose inference
engines. They have to support many architectures, different GPUs, different quantization formats,
multiple operating systems and a huge range of hardware. Flash-Next creates opportunities
for optimizations that are much more specific to this particular architecture. And that's where
projects such as EXL3, Trellis and Strata become interesting. EXL3 and Trellis represent the
more experimental side of Flash-Next inference. Instead of waiting for every optimization to make
its way into a general-purpose engine, developers can specialize around things such as aggressive
quantization, expert placement, memory management and highly optimized GPU execution. This is
actually fairly common in local AI. New architectures often appear first in specialized implementations
or forks. Once the techniques become stable and prove useful, some of those ideas can eventually
migrate into larger inference frameworks. Strata takes specialization in another direction.
Instead of simply asking how to fit as much of Flash-Next as possible into GPU memory,
Strata treats the GPU, CPU, system RAM and even SSD storage as different parts of the inference
system. The GPU handles components that are used constantly, along with frequently selected
experts. System RAM can hold the complete collection of experts, while the CPU calculates
experts that aren't resident on the GPU. Importantly, Strata tries to perform CPU and GPU computation
concurrently. That's different from traditional offloading, where the GPU can spend significant
time waiting for weights or computation coming from the CPU. With an MoE architecture,
only a subset of experts is selected for each token, creating an opportunity to divide that
work intelligently between processors. The enormous PLE lookup structure can also live on storage,
because only a tiny fraction of it is accessed for each token. Strata then combines this heterogeneous
execution with Flash-Next's multi-token prediction capability, allowing the model to
propose and verify multiple tokens rather than always progressing exactly one token at a time.
So these projects are really solving different levels of the same problem.
llama.cpp is trying to make Flash-Next a first-class architecture inside a broadly portable local
inference ecosystem. SGLang approaches it from the perspective of a high-performance inference
and serving framework, where GPU utilization, batching, throughput and production deployment
are especially important. Projects such as EXL3 and Trellis can experiment more aggressively
with specialized quantization and execution techniques. And Strata focuses heavily on exploiting
heterogeneous hardware, intelligently distributing different parts of Flash-Next across GPU memory,
system RAM, CPU compute and storage. Over time, these approaches may start to converge.
Techniques demonstrated in experimental implementations can eventually find their way
into mainstream inference engines. And that's what makes Flash-Next particularly interesting.
A dense model with more than a hundred billion parameters would require enormous computation
for every generated token. Flash-Next doesn't. Only a fraction of its parameters are active at
once. Its massive lookup structure doesn't necessarily have to live on the GPU.
Its expert routing creates opportunities for intelligent CPU and GPU scheduling.
And multi-token prediction provides another path toward faster generation.
So Flash-Next isn't simply another large model waiting for llama.cpp and SGLang to add compatibility.
It represents a different inference problem. And projects like llama.cpp, SGLang,
EXL3, Trellis and Strata are exploring different answers to the same question:
if only a small portion of an enormous model is needed at any given moment,
what's the smartest way to use the hardware we already have?