Unsloth has released a new 4-bit version of Qwen3.6 that it describes as up to 2.5 times faster. The release uses NVIDIA’s NVFP4 format, and it can run locally on a single Blackwell GPU. That immediately caught my attention, because the headline number sounds excellent, but data center benchmark figures don’t always map cleanly to a single consumer graphics card.
So I tested it on my own RTX 5090 workstation. I ran the 27B NVFP4 version of Qwen3.6 through vLLM, connected it to the Pi coding agent, compared it with a GGUF model running in llama.cpp, and tested multi-token prediction at different draft depths.
The short version is this: the NVFP4 model is genuinely faster on Blackwell hardware, but the most meaningful local speed improvement came from enabling multi-token prediction, also called speculative decoding. With the right settings, I reached 144 tokens per second locally while running a coding agent. That is a serious pace for a fully local setup.
There are important details behind that number. Hardware generation, inference kernel, model format, model architecture, and draft depth all affect the result. The point of this test was to separate the exciting headline from what actually matters when running local AI on one GPU.
What Unsloth Released
Unsloth takes large open models and creates optimized versions that are practical to run on local hardware. If you have downloaded a quantized model before, there is a reasonable chance you have used one of its releases.
This release is based on Alibaba’s Qwen3.6 family. The models are Apache 2.0 licensed, so the weights can be downloaded and run without an API bill or usage restrictions. Qwen3.6 is particularly interesting for local AI because it has strong coding capabilities, a native multimodal design, and a context window up to 256,000 tokens.
Unsloth published two distinct model shapes:
- Qwen3.6 27B: A 27-billion-parameter dense model.
- Qwen3.6 35B-A3B: A mixture-of-experts model with 35 billion total parameters and a smaller active subset for each request.
The difference matters. A dense model works like an office where every member of staff contributes to every question. Each generated token involves the whole group.
A mixture-of-experts model works more like a much larger office with a receptionist. The receptionist routes each question to the few specialists who are most relevant. The model can contain a large pool of expertise without activating every part of it for every token.
That expert routing becomes important because Unsloth applies its dynamic precision approach especially heavily to the expert layers. The model can use very low precision where it is safe, while keeping more sensitive parts of the network at higher precision.
The published model weights are available through the Qwen3.6 27B NVFP4 model page and the Qwen3.6 35B-A3B NVFP4 model page.
Why Quantization Makes Large Models Practical Locally
A language model is, at its core, a huge collection of numbers called weights. Every fact it has learned, every code pattern it recognizes, and every relationship between words exists in billions of numerical values.
When those values are saved to disk, they form a model checkpoint. Those large downloads on Hugging Face are the model’s weights written into files.
Quantization means storing those numbers with fewer bits. You can think of it as reducing the number of decimal places in a very long number. Each value becomes less exact, but the file becomes dramatically smaller. If the process is done well, the model retains most of its useful behavior.
Going down to 4-bit precision creates a significant memory saving. The 27B dense model can fit in around 24 GB of VRAM, while the 35B-A3B model can fit in around 32 GB. That puts both options within reach of a single high-end consumer GPU, such as the RTX 5090 in my test machine.
There is always a trade-off. Fewer bits save storage and memory bandwidth, but reducing precision too aggressively can make a model less reliable. The work in quantization is about finding the parts of the model that can tolerate lower precision and protecting the parts that cannot.
Every Generated Token Has Two Costs
To understand the NVFP4 speed claim, it helps to understand what happens when a model produces text. A prompt is converted into numerical representations. Those values pass through layer after layer of weights. The model performs an enormous number of multiply-and-add operations before it produces the next token.
Each token has two main costs:
- Memory cost: The GPU has to move model weights through memory.
- Math cost: The GPU has to multiply the activations and weights across the network.
Traditional 4-bit quantization helps the first cost. Smaller weights mean less data needs to move, so memory traffic drops. This is why quantized models have long been useful for local inference.
The second cost is where NVIDIA’s Blackwell architecture changes things. On previous GPU generations, a model may store its weights in 4-bit form, but the GPU often needs to expand those values before doing the multiplication. The weights stay small in storage, yet the math happens at wider precision.
NVFP4 changes that on Blackwell hardware. NVIDIA built support for native 4-bit multiplication into Blackwell Tensor Cores. The RTX 50 series and data center B200 GPUs can multiply 4-bit values directly. There is no need for the same expand-before-multiply step.
That is the key promise of NVFP4. It reduces the weight movement cost and can reduce the actual math cost. Four-bit no longer acts only as a storage trick. On the right hardware, it can change the computation itself.
There is an important limitation. NVFP4 is a Blackwell-only format. These quants cannot run on RTX 30 or RTX 40 series cards because those GPUs do not have the required native 4-bit multiplication hardware. For older cards, Unsloth’s dynamic GGUF releases are the more relevant option.
W4A4 Versus W4A16: The Accuracy Trade-Off
The naming around these model formats can sound more complicated than it is. W4A4 means the model uses 4-bit weights and 4-bit activations. The weights are the stored model values. The activations are the numbers flowing through the model while it processes a request.
W4A16 also uses 4-bit weights, but it retains 16-bit activations. That still provides the memory savings of 4-bit weights. However, it does not fully take advantage of native 4-bit math because the GPU is still multiplying with higher-precision activations.
NVIDIA’s own safer default for this Qwen3.6 quant is W4A16. That makes sense because activations are often where a model becomes sensitive to reduced precision. Pushing everything to 4-bit can affect accuracy.
Unsloth’s release takes a more nuanced route. Instead of choosing W4A4 or W4A16 for the entire model, it uses a dynamic layer-by-layer strategy.
How Dynamic NVFP4 Works
Dynamic NVFP4 means Unsloth spends precision where it matters and saves it where it is safe. The model does not force every component into the same low-precision format.
In the mixture-of-experts 35B-A3B model, the expert weights make up a large part of the total model. These specialist blocks are pushed into W4A4, where Blackwell can use native 4-bit multiplication.
More sensitive components remain at higher precision. These include attention layers, vision layers, and the final output layer. They remain in FP8 or full precision where necessary.
This is what makes the release dynamic. It is not a blunt choice between fast and accurate. It applies lower precision selectively, based on which parts of the model can handle it.
The 35B-A3B model comes in two relevant variants:
- Fast: Pushes expert layers to 4-bit more aggressively and is reported at 1.79 times faster.
- Standard: Keeps the last eight layers wider, is reported at 1.56 times faster, and gains a small improvement on MMLU Pro.
The performance figures from Unsloth are very compelling. On a B200, the W4A4 quant delivered around 6,800 tokens per second compared with around 2,400 tokens per second for NVIDIA’s W4A16 version in the tested setup.
However, there is a crucial word attached to those numbers: throughput.
Why Data Center Throughput Does Not Equal Local Speed
The 2.5 times figure measures server throughput. In the B200 test, a single GPU serves 128 users at once. At that scale, the GPU spends a large amount of time doing math. Native 4-bit multiplication has a major impact because the system is compute-bound.
A local setup is different. One person is using one GPU. In that case, the bottleneck is often memory bandwidth rather than raw multiplication. The GPU spends much of its time moving weights rather than calculating them.
That does not make NVFP4 irrelevant for local inference. It simply means I should not expect a consumer RTX 5090 to reproduce a data center throughput benchmark. The local decoding gain is more modest because the workload has changed.
This distinction is one of the most useful lessons from the test. Benchmark headlines need context. A format may produce a massive benefit for a busy inference server, while a single-user workstation sees a smaller gain. Both results can be true at the same time.
Two Settings That Matter for Local Qwen3.6 Performance
For local inference, I found two main levers worth focusing on: the kernel and multi-token prediction.
Use a Kernel That Supports W4A4
A kernel is the low-level program that runs the model math on the GPU. It sounds technical, but the practical implication is straightforward: the wrong kernel can prevent the hardware from using the fastest available path.
Older kernels, including Marlin, do not support W4A4. For Blackwell NVFP4 inference, the relevant choices are QTDSL or FlashInfer. These kernels allow the GPU to use the format properly.
If the model format, GPU, and kernel do not align, the potential hardware advantage may simply sit unused.
Enable Multi-Token Prediction
Multi-token prediction, or MTP, delivered the biggest local speed improvement in my test. It is included in these checkpoints and acts like a form of speculative decoding.
Instead of generating one token at a time, a drafting component proposes several future tokens. The main model then checks those proposed tokens in one pass. Verification is much cheaper than generating each token independently.
The core idea is simple:
- Writing each token is expensive.
- Checking several drafted tokens together is comparatively cheap.
- Accepted drafts allow the model to move ahead faster.
The main control is called draft depth. This determines how many tokens the drafter proposes ahead of time. I tested depths from one to six. Two is a common default, but this setting is hardware-dependent. It is worth testing rather than assuming a default will be optimal.
My RTX 5090 Test Setup
I ran the comparison on a local workstation with one RTX 5090 GPU. I used two engines:
- vLLM running the Qwen3.6 27B NVFP4 quant.
- llama.cpp running a GGUF quant for comparison.
I used multi-token prediction with a draft depth of four for the main comparison. Both configurations worked from the same physical machine and were measured under the same basic local conditions.
The resulting numbers were:
- Qwen3.6 27B NVFP4 in vLLM with MTP depth 4: 144 tokens per second.
- Qwen3.6 GGUF in llama.cpp with MTP depth 4: 116 tokens per second.
That is a clear local advantage for NVFP4 on the RTX 5090. It is not the full 2.5 times gain from the B200 throughput test, but it is still meaningful. At depth four, the Blackwell-specific format pulled ahead of the GGUF option.
The biggest change came when I compared MTP against non-MTP runs.
- llama.cpp GGUF without MTP: 69 tokens per second.
- llama.cpp GGUF with MTP depth 4: 116 tokens per second.
- NVFP4 without MTP: 64 tokens per second.
- NVFP4 with MTP depth 4: 144 tokens per second.
On the NVFP4 model, enabling MTP increased generation speed from 64 tokens per second to 144 tokens per second. That is more than a twofold increase from one setting.
For a single developer running a local model on one GPU, that is the practical result worth paying attention to. Native 4-bit math helps. Speculative decoding can transform the feel of the system.
Draft Depth Produced an Interesting Result
One part of the test surprised me. The GGUF model was faster than NVFP4 at draft depths of one, two, and three. The NVFP4 model became substantially stronger at depths four, five, and six.
This is exactly why local benchmarks matter. The best setting depends on the GPU, model format, engine, kernel, and MTP implementation. A lower draft depth is not always better. A higher one is not always better either.
On my RTX 5090, depth four was the point where NVFP4 began to show its stronger advantage. That gave me the 144-token-per-second result.
The lesson is not that depth four will always be the answer. The lesson is that draft depth is a real performance dial. Test one through six on the actual hardware that will run the workload.
Putting Qwen3.6 NVFP4 to Work in Pi Agent
Speed benchmarks are useful, but I wanted to see the model do actual coding work. So I ran it through the Pi coding agent inside a Docker container.
I gave the agent a contained task: build a small command-line expense tracker in Python. The application needed an SQLite data store, a set of commands, seeded data, and tests. I asked it to write pytest coverage and iterate until the test suite passed.
The model generated at roughly 144 tokens per second while Pi worked through the project. Files and folders appeared in the workspace. The agent wrote the application, ran tests, encountered failures, made corrections, and repeated the process.
The GPU was fully utilized by vLLM during the task. The agent progressed through its test failures and continued correcting the implementation. In the end, all 40 tests passed.
I then checked the result directly. The application created an SQLite database with categories and expenses tables. It could list seeded expenses, filter them by category, calculate totals, and add a new expense.
For example, the housing category total came to 535, matching the three relevant transactions in the database. After adding a new Coffee Break expense, the new entry appeared in the output after refresh.
This is a deliberately simple command-line application. It does not represent the complexity of a large production codebase. It does show the full local loop working: model generation, tool use, file creation, test execution, correction, database use, and final validation.
Why Fully Local AI Matters Here
The technical speed result is useful, but local execution has another important benefit. Everything stays on the machine. The prompts, source code, tests, database data, model outputs, and agent work remain local.
There is no per-token charge from a hosted model provider. There is no need to send project context to an external API for this workflow. Once the hardware and electricity are in place, generation can continue without metered token costs.
That does not mean local models replace every hosted model or every coding workflow. It does mean that a high-end single-GPU workstation can now run a capable coding agent at a pace that feels far more practical than many people expect.
Practical Guidance for Running Unsloth Qwen3.6 Locally
Based on this test, I would use the following checklist before drawing conclusions from any Qwen3.6 NVFP4 benchmark.
- Confirm the GPU generation. NVFP4 requires Blackwell hardware. RTX 50 series and B200 GPUs support the native 4-bit multiplication path.
- Choose the right model format. Use NVFP4 on Blackwell. Use a dynamic GGUF release if you are running older hardware.
- Use a compatible kernel. QTDSL or FlashInfer are relevant for W4A4. Older kernels such as Marlin do not support this route.
- Enable MTP. The checkpoint includes multi-token prediction support, and it produced the biggest local speed increase in my tests.
- Benchmark multiple draft depths. Test values from one to six. The optimal point may change by engine, model format, and hardware.
- Separate throughput from single-user decoding speed. Data center numbers can be valid without predicting the exact local result.
- Validate output quality on the task you care about. The published benchmarks are knowledge benchmarks. Coding performance deserves coding-specific testing.
For repeatable comparisons across engines and MTP settings, I published the benchmark setup in the Qwen3.6 arena repository. I also used the open-source Pi coding agent for the local coding test.
What the 2.5× Claim Means in Practice
Unsloth’s 2.5 times faster headline is real in its stated context. On a B200 handling high-throughput workloads, W4A4 native 4-bit math can produce a major performance improvement over W4A16. That is a compute-heavy server scenario, and NVFP4 directly targets that kind of math cost.
My single RTX 5090 setup told a more nuanced story. NVFP4 beat the GGUF comparison at the right MTP depth, reaching 144 tokens per second against 116. Yet the major local gain came from multi-token prediction, which more than doubled NVFP4 speed from 64 to 144 tokens per second.
The combination is what makes this release interesting: a capable open Qwen3.6 model, 4-bit weights small enough for one high-end GPU, Blackwell-native W4A4 math, and MTP built into the checkpoint.
For anyone building a local coding workflow on an RTX 5090, the useful configuration is clear. Run the NVFP4 quant through a compatible engine and kernel. Enable MTP. Test draft depth carefully. Then measure the result on real work rather than relying only on a single headline number.
