Nine model builds and four runtimes through the same tests: speed, long context, quality and real agent tasks. The winner for my agent: Qwen3.8 Flash Next at 2.75 bits on sushi.
model builds
9
runtimes
4
fastest writing
119 tok/s
recall at 199K tokens
12/12
Head to head on my agent's workload
Same three requests for each model: a 600-word answer in French (3 runs, median shown), a 42K-token code dump with one fact hidden in the middle, and 8 tool-calling cases. Every model found the hidden fact.
Writing speed, tokens/s, higher is better
Qwen3.8 Flash Next, 2.75 bpw · sushi119.1Ornith 1.5 122B, 4 bit · oMLX104.7Qwen3.8 Flash Next, 4 bit · oMLX93.2Qwen3.8 27B dense, 4 bit · oMLX74
Reading a 42K-token prompt, tokens/s, higher is better
Qwen3.8 Flash Next, 2.75 bpw · sushi2339Ornith 1.5 122B, 4 bit · oMLX1786Qwen3.8 Flash Next, 4 bit · oMLX1356
The 27B run came from the server's prompt cache, so its reading speed isn't a real measurement and is left out. Tool calls: all four passed 8 of 8 in these runs. Download every number on this page (JSON).
Speed as the prompt gets longer
Four prompt sizes from 1K to 128K tokens, each made unique so no cache helps, 2 runs per size. The 2.75-bit build with MTP barely slows down: it loses 3% of its writing speed between 1K and 128K. The 4-bit build reads prompts fastest but needs a lot more memory.
Writing speed, French prose, tokens/s
Flash Next 2.75 bpw, MTP · sushi
Flash Next 2.75 bpw, DFlash · sushi
Flash Next 4 bpw, MTP · sushi
Flash Next 3.3 bpw, MTP · mlx-serve
DeepSeek V4.1 Flash 1.75 bpw · sushi
The numbers
1K
16K
64K
128K
Flash Next 2.75 bpw, MTP · sushi
98.5
100.7
91.8
95.4
Flash Next 2.75 bpw, DFlash · sushi
95.3
88.7
86.9
83.7
Flash Next 4 bpw, MTP · sushi
103.1
103.2
99.6
90.9
Flash Next 3.3 bpw, MTP · mlx-serve
99.2
93.3
82.5
78.8
DeepSeek V4.1 Flash 1.75 bpw · sushi
38.8
39.2
39.2
38.7
Reading speed (prefill), tokens/s
Flash Next 2.75 bpw, MTP · sushi
Flash Next 2.75 bpw, DFlash · sushi
Flash Next 4 bpw, MTP · sushi
Flash Next 3.3 bpw, MTP · mlx-serve
DeepSeek V4.1 Flash 1.75 bpw · sushi
The numbers
1K
16K
64K
128K
Flash Next 2.75 bpw, MTP · sushi
1,652
2,372
2,297
2,252
Flash Next 2.75 bpw, DFlash · sushi
1,649
2,312
2,264
2,192
Flash Next 4 bpw, MTP · sushi
1,610
3,401
3,237
3,150
Flash Next 3.3 bpw, MTP · mlx-serve
563
1,831
2,128
2,046
DeepSeek V4.1 Flash 1.75 bpw · sushi
216
668
663
646
Writing code is faster than prose on every build (100–145 tokens/s), because code is easier for the MTP head to guess. DeepSeek V4.1 Flash streams its experts from the SSD, which is why it stays near 40.
Quality: almost everything at the ceiling
Tool calls and auto-checked answers were repeated 4 times. Code runs against hidden tests, maths answers are matched against a key, and the long-context test hides 5 notes in a 199K-token prompt, one of them a later correction of another.
Build
Tool calls
Checked answers
Code, hidden tests
Maths
Recall at 199K
French grammar
French fixes
Flash Next 2.75 bpwsushi
29/32
16/16
54/54
48/48
12/12
28/30
8/8
Flash Next 4 bpwsushi
30/32
16/16
54/54
48/48
12/12
27/30
8/8
Flash Next 3.3 bpwmlx-serve
32/32
16/16
54/54
47/48
12/12
26/30
8/8
DeepSeek V4.1 Flash 1.75 bpwsushi
27/32
14/16
–
–
–
–
–
27B fine-tune "Turbo 735"llama.cpp, 4 bit
–
–
36/36
31/32
–
28/30
5/8
27B fine-tune "Jumbo 747"llama.cpp, 4 bit
–
–
35/36
32/32
–
27/30
4/8
Bold means a perfect score. The 3.3-bit build on mlx-serve was the only one with 32/32 tool calls, so on tools my pick is not the best. The two community 27B fine-tunes ran a shorter code and maths set (2 samples instead of 3): strong there, but they fixed only half of the French sentences.
Real agent tasks
Benchmarks are one thing. These are two jobs I handed to Hermes Agent with the local model in charge, from one prompt, with no help.
Build a complete shop website
A watch shop in plain HTML, CSS and JavaScript: catalogue, filters, cart, checkout, responsive, accessible. Up to 150 steps, one hour. Graded on a 100-point rubric.
One prompt in French
Hermes plansup to 150 steps
Writes 24 filesHTML, CSS, JavaScript
Checked in Chromefeatures, mobile, Lighthouse
96/100 in 11.6 min
STQ 2.75 bpw sushi
iQ 3.3 bpw mlx-serve
Total, out of 100
96
97
Features, /40
40
40
Mobile, /25
25
24
Accessibility, /15
15
15
Design, /20
16
18
Time to build
11.6 min
20.4 min
Both scored 100 on Lighthouse mobile accessibility, best practices and SEO. The 2.75-bit build finished in about half the time.
Fill a PDF form and sign it
A 6-page insurance form, 6 values to enter, and a handwritten signature to cut out of a screenshot with a real transparent background and place in 3 boxes. 2 runs per build.
8 of 8 passed the automatic check. The scorer only checks values, image placement and transparency, not how good the signature looks.
French writing, judged blind
Most of what my agent writes is in French, so I graded 12 everyday writing tasks with the two finalists shuffled under neutral labels. Result: 98 for the 2.75-bit build against 97.5 for the 3.3-bit build, out of 120. The 2.75-bit build won 7 tasks, lost 3 and tied 2. A tie, honestly.
Task
2.75 bpw · sushi
3.3 bpw · mlx-serve
Chat message about a backup
9
8.5
Formal letter to an administration
7
7
Explain something to a beginner
7
9
Summarise server logs
9
8
Formal rewrite
9
8
Technical translation
9
9
Apology to a customer
8
7.5
Daily briefing
7
8.5
Social media post
8
9
Steps for an elderly person
8.5
7.5
Reply to a complaint
7.5
7
Short opinion piece
9
8.5
Scores out of 10, the winner of each task in bold. One grader, one sample per task.
Temperature 0.6 or 1.0?
Same model, same tests, only the sampling temperature changed. 0.6 matched or beat 1.0 everywhere, so it's my default now.
Test
T = 0.6
T = 1.0
Tool calls, 4 runs
29/32
28/32
Harder code tasks
18/18
16/18
Maths
48/48
48/48
How I run them
Through the real API. Every test sends streaming requests to the same OpenAI-compatible endpoint my agent uses, and times them from the client.
Answers are checked by code where possible. Hidden unit tests for code, an answer key for maths, exact checks for tool names and arguments.
Caches are defeated. Each long prompt is made unique, and runs that hit the cache anyway are thrown out.
Several runs. 2 to 4 per test depending on how long it takes. I show the median or the mean, never the best run alone.
The gaps stay visible. If a run was aborted or a number can't be trusted, it says so below.
What these numbers don't tell you
Speed tests ran on sushi 1.4.1. I now run 1.5.1 and haven't re-measured yet.
The agent tasks are one or two runs each, and the design score and the blind French grades come from a single grader.
Some result files don't record the build they ran; where I inferred it from timing and file names, the raw data says so.
Every test ran on one machine. A Mac with less memory would rank these builds differently.
The open models people download most
Not measured by me yet: these are estimates from each model's published config, at 4-bit with an 8K context.
I benchmarked DSpine speculative decoding on a Mac Studio M5 Ultra. Qwen3.8-27B goes from 43 to 134 tok/s on code with a 1.7 GB draft. Full idle numbers.
_front.jpg){kind=link}