oQ in oMLX: what I found about Qwen3.8 quants and where my measurement disagreed with expectations

2026-10-11Русская версия

A month ago I benchmarked engines for a local Qwen3.8-27B. Since then oMLX has improved its oQ quantizer, dozens of oQ builds of Qwen3.8 have appeared on Hugging Face, and in 0.7.1.dev1 oMLX learned to serve decision models. I collected what is known and re-measured on my M4 Max 48 GB: oQ4e against plain 4-bit, with and without MTP, and the clef decision model on 600 labeled URLs.

How it started

In September I benchmarked eight engines for a local 27B (in Russian) and ended up benchmarking my laptop. What won back then was not the engine but the quant and how much memory was left for the system. Since then enough has changed around Qwen3.8 on the Mac to make a second pass.

I didn’t collect it by hand. The qwen-3-8 topic is run by my research CLI pro2you: it runs searches from approved queries, downloads pages, pulls facts with quotes out of them and rebuilds the summary once per round. This round I added queries about oMLX 0.7.1 and oQ quantization, and taught the tool to use the Hugging Face API: it now reads a model page not as HTML but as a card with the license, parameters, files by format and the list of quants.

The round’s result: 59 queries, 161 new URLs, 261 downloaded pages, plus 493 facts. 5160 facts on the topic in total. Below is only what has a primary source. Where a number is someone else’s and nobody has re-checked it, I say so.

What oQ is

oQ is the quantizer built into oMLX. The output is a regular MLX model in safetensors, readable by both mlx-lm and mlx-vlm, no custom loader needed (oQ documentation).

The main difference from plain mlx-lm 4-bit is mixed precision. Before writing, oQ runs calibration prompts, measures how much each layer suffers from quantization and adds bits where they help most. The total size stays around the chosen level.

There are also intermediate oQ2.5, oQ2.7 and oQ3.5 for MoE. For dense models the documentation recommends starting with oQ3e: at 2 bits a dense model loses noticeably more than a MoE.

The letter e in the name means oQe. Same bit allocation, same size and speed, the rounding is different. oQe first collects an importance matrix (imatrix, as in llama.cpp) and protects the channels that carry the main signal when rounding.

What oQ leaves alone or keeps at higher precision:

• the output head and embeddings - 8 bits, if the budget allows;
• MoE routers - full precision;
• vision and audio encoders - not quantized;
• norms and recurrent state parameters (SSM, linear attention) - not quantized;
• MTP heads with Preserve MTP weights - kept, hence the -mtp suffix in names.

The -fp16 suffix means the unquantized weights and scales are stored in float16, not bfloat16.

What changed in quantization

The reason for this pass was news that the quantization method changed in 0.7.1.dev1. The oQ documentation has a section on it: oQe got a refit of scales and biases by weighted least squares, PR #4385. The refit is repeated several times, and the result is kept only if the error goes down.

This is the oMLX author’s measurement: MMLU 1000, Winogrande 300, MBPP 300 and GSM8K 300, 1900 questions, greedy, no thinking. According to him, KL divergence to the original drops by 12-15%.

Two remarks. First: oQ4e “after” scores above the original for both models. That means a half-point difference on 1900 questions is within noise, and the table should be read as “oQ4e loses almost nothing”, not as “the quant beats the original”. Second: the #4385 refit shipped in 0.7.1.dev1, the release notes say so directly. They also add a caveat: only new conversions change. oQe builds made earlier don’t get the refit.

An earlier version of the same comparison, still without the refit, exists for Qwen3.6-27B, the closest relative of our model: original 85.50 (55.6 GB), mlx-lm 4-bit 84.25 (16.1 GB), oQ4 84.50, oQ4e 84.81 (16.7 GB). So oQ4e wins back more than half of what plain 4-bit loses, at the cost of 0.6 GB.

Which oQ builds of Qwen3.8 are on the Hub

Download numbers are from the Hugging Face API, the count for the last 30 days as of 2026-10-11.

Jundot is the author of oMLX, these are the official builds. A community has already grown around them: thederby published the full oQ4e-oQ8e line for 27B, notnaurw - three oQ5e variants with different bit allocation strategies (SAGE-PATHS, SAGE-ARCS, SAGE-CREST), scottlowry - oQ builds of the fine-tuned Swift-1.5 and ThinkingCap. For SAGE-CREST the author reports KL at 128K context 25% better than uniform 6-bit (card). That’s his measurement, I didn’t re-check it.

Almost all builds keep the MTP head and the vision encoder. For SAGE-PATHS someone calculated that the encoder weighs about 0.92 GB in bf16 and is dead weight in text-only work (card). oQ has a Text Only option for that.

A separate note on Flash-Next. It’s a 125B MoE with 6B active, plus a 51B n-gram table and a 4B MTP (umixer card). The most downloaded oQ build overall is exactly this one. Figures of 45 tok/s on M4 Max and 25 tok/s on M2 Ultra float around articles for it, but I found no primary source with a measurement.

My measurement: M4 Max 48 GB

The other people’s tables below are mostly about M5 and 128 GB machines. I have an M4 Max with a 40-core GPU and 48 GB, of which Metal gives out about 40 GB. So I re-measured the main thing myself, with the built-in oMLX 0.7.1.dev1 benchmark, the same one that writes to the omlx.ai database.

I compared three variants of one model on one engine version:

• mlx-community Qwen3.8-27B 4bit, 14.8 GB - plain uniform 4-bit;
• thederby/Qwen3.8-27B-oQ4e-mtp, 17.0 GB - oQ4e built on 0.7.1.dev1, with the new refit. The build’s author writes that the refit lowered KL to the original from 0.0428 to 0.0358 compared with earlier oQ4e builds of this model;
• the same with MTP on.

Context 1k, 4k, 16k and 32k, 128 generated tokens, two text sets (prose and Python), each run twice. Before measuring I unloaded Docker and extra models: in the first attempt the oMLX memory guard cut off long contexts.

Prose (novel_en), mean of two runs, spread in parentheses:

Time to first token at 32k is the same for all three: 140-146 seconds.

What follows from this.

The quant itself gives no speed. Without MTP oQ4e is 2-6% slower than uniform 4-bit in generation, prefill is the same. The file is 2.2 GB bigger, and generation on a Mac is bound by memory bandwidth. Part of the difference may come from heat: 4-bit ran first in each series, and by the time oQ4e ran the chip was already going into thermal state 2. oQ4e is chosen for quality, not for speed.

Speed comes from MTP. On prose plus 20-60%: 34-40 tok/s instead of 25-28. At 32k the gain drops to 23%. The price is about 2.3 GB of extra memory. On 48 GB that’s noticeable: the oMLX memory guard stopped the first MTP run at 32k, it was 0.27 GB short.

On code the MTP numbers are not good for conclusions. On Python I got 50-55 tok/s at 1k, 40 and 82 tok/s in two runs at 16k and 101-106 tok/s at 32k. Only 128 tokens are generated, and on a long chunk of Python code the model seems to continue what was already in the prompt. The MTP draft then guesses right almost every time. Without MTP both builds give 23-28 tok/s on the same texts, as on prose.

Qwen3.8-Flash-Next doesn’t fit into 48 GB in any oQ. oQ3 weighs about 84 GB, oQ4e even more. All the nice Flash-Next numbers below are for machines with 128 GB and up.

For comparison: the oMLX run database

oMLX has an open database of measurements: omlx.ai/benchmarks. Users run the built-in benchmark and submit the result; as of 2026-10-11 there are 524,164 records. The site’s table has the chip, memory, model, quant, context length, prefill speed (PP) and generation speed (TG). The oMLX version and time to first token are not in the common list, only on the page of an individual run.

Below are all Qwen3.8 runs that made it to the first page of the model filter. Each row is one run by someone else, no repeats and no spread. Chip and memory are given in the first row of a group, the rows below are runs of the same machine at another context.

Qwen3.8-27B

The M4 Pro 16k and M4 Max rows come from the pages of individual runs (M4 Pro, M4 Max), where the version is visible too: oMLX 0.5.7 for M4 Max. On M4 Pro at 16k the first token arrived after 2.9 minutes.

Qwen3.8-Flash-Next

What I take from this.

Prefill on M5 grew several times over, generation didn’t. MLX 4bit at the same 4k: M4 Max gives 249 tok/s of prefill, M5 Max - 1007, four times more. Generation is 27.4 against 32.0, plus 17%. Prefill is bound by compute, generation by memory bandwidth, and M5 gained in the former. For an agent that reads a long context every time, this matters more than tok/s in a chat. Caveat: these are different people and most likely different oMLX versions.

oQ4e-mtp against plain 4bit - 2-2.7 times faster in generation. On M5 Max at 16k: MLX 4bit 30.6 tok/s, oQ4e-mtp 73.0 and 83.6 in two different runs. This isn’t a pure quant effect: -mtp builds run speculative decoding, and that’s what gives the main gain. How much of the difference is due to oQ and how much to MTP can’t be separated with this database.

Flash-Next barely notices context length. On M5 Max with oQ4e-mtp generation stays within 62-85 tok/s from 1k to 195k tokens, prefill above 2000 tok/s. The dense 27B on the same chip with oQ4e-mtp drops to 47.6 at 128k. This matches the architecture: Flash-Next has 6B active parameters and three quarters of its layers on linear attention (Gated DeltaNet), the KV cache grows only for 12 layers out of 48.

oQ8e Flash-Next on 128 GB is a trap. 7.5-11.3 tok/s against 70 for oQ4e on the same M5 Max. By my estimate, an 8-bit build of a 180B-parameter model together with the n-gram table doesn’t fit into 128 GB, and oMLX pages weights in. The database doesn’t say so, the conclusion is mine.

The bottom of the 27B range is on the edge. M4 Pro and M5 Pro with 48 GB give 8-15 tok/s, and OptiQ on M5 Pro 16c is slower than MLX 4bit on M4 Pro. Why, you can’t tell from a database row.

There is a counterexample for MTP too. At halfpennymac on M4 Pro, MTP in llama.cpp slowed generation by 26%, from 10.7 to 7.9 tok/s, even though the draft was accepted 65% of the time. So “-mtp is always faster” is about oMLX on chips with bandwidth to spare, not a rule.

My measurement: a decision model through oMLX

Here I checked it myself. In pro2you, whether a found page is relevant is decided not by a chat model but by a decision model: it reads the text once and returns a probability for each answer option, generating nothing. I use clef-flash from Cloudflare (Qwen3.5-9B plus a decision head) through /v1/systemone in llama.cpp, quant GGUF Q4_K_M.

oMLX 0.7.1.dev1 added support for the same models through the same /v1/systemone (#4315). The release note says separately that oQ keeps the clef head intact, and there are ready-made Jundot/clef-oQ4e and Jundot/clef-flash-oQ4e.

The set: 600 URLs labeled by three judge models (qwen3.8-27b, gemma-4-31b, gpt-oss-120b), the correct answer is the majority decision. One question: “is the page on topic”. Requests go sequentially, on M4 Max 48 GB.

The most interesting part is not the AUC but the second-to-last row. With llama.cpp the probabilities are squeezed into a band from 0.29 to 0.75: the model is “almost never sure”. With oQ4e they spread from 0.01 to 0.96. It looks like this is the preserved head: in GGUF Q4_K_M it gets quantized along with everything else, and the scale gets smeared. I can’t check this directly - I’d need to compare the same head in fp16 inside a GGUF, and I don’t have such a build.

The practical consequence: the threshold from the old config can’t be carried over. At 0.5 clef-flash-oQ4e almost never errs on “yes” (precision 0.991), but misses every fifth on-topic page (recall 0.800). The threshold has to go down:

For a findings filter it matters more to me not to lose an on-topic page than to let an extra one through, so I take 0.1: recall is higher than llama.cpp at 0.5 (0.921 against 0.868), with comparable precision (0.900 against 0.924).

The 27B version of clef adds almost nothing: AUC 0.986 against 0.982, and on unanimous URLs it never made a mistake. But each request takes four times longer. For a filter that runs hundreds of pages per run, that’s a bad trade.

clef-flash has the same speed on both engines, and neither oMLX model returned a single 500 error in 600 requests. llama.cpp did have them, so the config had retries.

What I didn’t find

• Nobody has a task-quality comparison of oQ4e and mlx-lm 4-bit for Qwen3.8-27B. There’s KL from the build’s author and tasks for Qwen3.6-27B from the oMLX author. I haven’t done my own quality measurement yet, only speed.
• The 4-bit vs oQ4e comparison in the omlx.ai database mixes hardware, versions and MTP, so the “for comparison” table is a reference, not a conclusion.

What I took for myself

• The decision model in pro2you moves to clef-flash-oQ4e in oMLX with threshold 0.1, llama.cpp stays as a fallback.
• bge-m3 in oMLX returns the same vectors as llama.cpp, cosine 1.0000 on the test texts, so the embedding cache survives the move.
• On M4 Max 48 GB for Qwen3.8-27B I take oQ4e with MTP: 34-40 tok/s on prose instead of 26-28, and there’s still memory headroom up to 32k context.
• For the next pass: oQ4e quality against mlx-lm 4-bit on my tasks with the oMLX built-in accuracy benchmark, with the official chat template. And EmbeddingGemma 2, which oMLX 0.7.1.dev1 already supports.