Setup
An M5 Ultra with 96 GB runs Qwen3.8 Flash Next on sushi, my agent, and about 40 containers. Here is where the memory goes and why I chose sushi.
A Mac Studio with the M5 Ultra and 96 GB of unified memory. The CPU and GPU share that memory, which is what makes a 180-billion-parameter model practical at home: no graphics card with its own small memory to squeeze it into.
The model server takes about half the memory. The rest runs my other services: a second model server for smaller models, about 40 Docker containers, and the usual apps. This is a snapshot from 9 November, not a plan, and it's tight: only a few GB are free.
The attention cache figure is an estimate from sushi's own memory check; the rest is measured. The lookup table and the SSD cache are read from disk as needed and don't count against memory.
sushi is an open-source model server for Apple Silicon, a fork of mlx-serve with its own low-bit format. I tried four runtimes on the same hardware (all the numbers). sushi won on the things my agent does all day: long prompts, fast answers, the same prompt prefix over and over.
95.4tokens/s
writing French with a 128K-token prompt. mlx-serve: 78.8.
+29%writing speed
against oMLX on the same model, and 2,412 vs 1,356 tokens/s reading a long prompt.
44.9GB of weights
at 2.75 bits, against 50.6 GB at 3.3 bits. Blind-graded French: a tie.
96%of prompt tokens
came from the cache on the SSD instead of being computed again.
68%of guesses kept
The model drafts the next tokens and checks them in one pass. Same output, faster.
3APIs, one port
OpenAI, Anthropic and Ollama, with tools, vision and JSON schema.
Where it lost: the 3.3-bit build on mlx-serve passed 32 of 32 tool calls against 29 for sushi's 2.75-bit build. Counters are from sushi's metrics on 9 November.
One command, started at login and restarted automatically if it ever stops.
| Flag | What it does |
|---|---|
--model Qwen3.8-Flash-Next-Sushi-STQ-2.75bpw | Load this one model folder: Qwen3.8-Flash-Next packed in Sushi's own 2.75-bit format. |
--host 127.0.0.1 | Listen on this Mac only. Nothing on the network can reach it. |
--ctx-size 262144 | Allow prompts up to 262,144 tokens (256K), the model's native context. |
--config-overrides (rope factor 1.0, 262K positions) | The pack ships with a static YaRN x4 stretch for 1M context. Setting the factor to 1.0 switches it off and caps context at the native 256K, because static YaRN hurts quality on short texts (plist note). |
--max-tokens 32000 | If a client does not say how long the answer may be, allow up to 32,000 tokens (room for long reasoning). |
--kv-quant 8 | Store the attention memory (KV cache) in 8-bit instead of 16-bit: half the RAM per token of context. |
--mtp-head-kv-quant | Also store the speculative MTP head's own KV cache in 8-bit (by default it stays 16-bit). |
--temp 0.6 | Default sampling temperature. 0.6 beat 1.0 on tool calls, hard evals and a signature task (plist note; temp-t06.json vs temp-t10.json). |
--top-p 0.95 | Sample only from the most likely tokens that add up to 95% probability (Qwen's recommended value). |
--top-k 20 | Never consider more than the 20 most likely next tokens (Qwen's recommended value). |
--max-concurrent 4 | Size the queue for 4 parallel requests; requests are batched together on the GPU. |
--prefix-cache-disk 20GB | Save already-processed prompt prefixes to the SSD (up to 20 GiB) so a repeated system prompt or conversation is restored, not recomputed, even after a restart. |
--prefix-cache-mem 2GB | Keep up to 2 GiB of the hottest prefixes in RAM as well, for instant reuse. |
(default) MTP speculative decoding | Not a flag here because it is on by default for Qwen: the model's built-in multi-token-prediction head drafts several tokens ahead and the main model checks them in one pass, plus prompt-lookup drafts from the n-gram table. Same output, more tokens per second. |
Nothing outside the Mac can reach the model server. Everything that needs a model goes through it locally.
The agent goes straight to the model
Everything else goes through a router
Every alternative ran on this Mac, through the same tests.
| Runtime and model | Result | Why I moved on |
|---|---|---|
| mlx-serve 26.11.1 (sushi's upstream) + Qwen3.8-Flash-Next iQ 3.3bpw | Decode 99/143 tok/s (prose/code) at 1K, 79/100 at 128K; prefill 563/916 at 1K. Tools 32/32 and quality 16/16 at temp 0.6; blind writing 97.5/120; French MCQ 26/30. | Slower at long context and on short-prompt prefill, ~6 GB more weights. Quality was a tie (it even scored better on tools: 32/32 vs 29/32). |
| oMLX 0.7.3 + Qwen3.8-Flash-Next oQ4e (4-bit, MTP) | Decode 93 tok/s; 41.7K-token prompt: prefill 1,356 tok/s, decode 77 tok/s; tools 8/8. | Clearly slower than sushi on the same model (120 / 2,412 / 119.5). |
| oMLX + Ornith-1.5-122B oQ4e MTP | Decode ~105-110 tok/s, long-prompt decode 79-84 tok/s, tools 7-8/8. | 68 GB of weights; together with Sushi's 50 GB it does not fit in 96 GB, so it was switched off on 2026-11-03. |
| oMLX + Qwen3.8-27B 4-bit (dense) | Decode 73 tok/s, long-prompt decode 56 tok/s, tools 8/8. | A dense 27B is much slower than the sparse Flash-Next MoE. Still kept as a secondary model. |
| llama.cpp llama-server, GGUF Q4_K_M 27B fine-tunes (two community merges) with MTP draft, 32K context, q8_0 KV | Decode median 52 and 64 tok/s (MTP draft acceptance ~0.81); hard code 24/24 and 23/24; French MCQ 28/30 and 27/30 but fixes only 5/8 and 4/8. | Half sushi's decode speed, worse on the French fix task, tested only at 32K context. |
| sushi + DeepSeek-V4.1-Flash STQ 1.75bpw, experts streamed from SSD | Decode 37-41 tok/s, prefill 216-786 tok/s; tools 27/32, quality 14/16. | Streaming from SSD makes it ~3x slower, with lower tool scores. |
| sushi + Qwen3.8-Flash-Next 4bpw pack | Similar decode (91-142 tok/s), faster prefill (~3,150-3,800 tok/s); tools 30/32; French MCQ 27/30. | 63.9 vs 44.9 GiB of weights for no quality gain ('ties 4bpw on every eval, 14 GB less RAM'). |
| sushi with DFlash2 drafter instead of MTP | Prose decode 84-95 tok/s vs 92-101 with MTP; code about equal. | MTP was 7-13% faster on prose. |
The longer write-ups go here. The first one goes to the RSS feed the day it's out.
More: Benchmarks · Workflows · Notes · all posts
Photos via Wikimedia Commons:Paul Hudson (CC BY 2.0), Paul Hudson (CC BY 2.0).
