Measured results, setup by setup
Every number below was measured on a real machine, most of them by the people who own it, and posted on GitHub with its log. Each table is one setup: its GPU (or GPUs), the system it ran on, and the models tried there.
- Prompt
- prompt processing (pp): how fast a prompt is read, in tokens a second, at the prompt length shown.
- Generation
- token generation (tg): how fast the answer is written, in tokens a second; one token is about three quarters of a word.
- With drafting
- generation with the model's draft head (MTP, DFlash2 or Gemma's drafter), which gives exactly the same text.
Models are Qwen3.8 27B GSQ-RCO IQ3_S unless named; the others are the files the launcher downloads. Numbers move with each release, so every row says which one. Setups we have not measured are the most useful thing you can add: how to measure yours.
Core Ultra 5 125H laptop
Arc graphics (Xe-LPG, 8 Xe-cores, DP4a, no XMX) · 32 GB RAM · Windows 11 · the laptop inferzero was built on · bench/website_bench.ps1, 2026-11-09
| Model | Prompt 8K | 32K | 131K | Generation | With drafting |
|---|---|---|---|---|---|
| Qwen3.5 4B | 896 | 705 | 366 | 21.0 | 35.8 |
| Gemma 4 E4B | 707 | 494 | 222 | 19.1 | 41.8 |
| Qwen3.5 9B | 487 | 435 | 278 | 11.8 | 21.6 |
| Qwen3.6 35B-A3B experts on the SSD | 851 | 637 | 324 | 21.6 | 38.1 |
| Gemma 4 26B-A4B experts on the SSD | 232 | 124 | 44 | 19.8 | – |
| Qwen3.8 27B | 135 | 125 | 81 | 5.1 | 12.9 |
| Qwen3.8 Flash Next 52 GB of experts on the SSD | 205 | 174 | 146 | 6.3 | 7.6 |
Generation over 256 tokens. Against llama.cpp on the same laptop and files (2026-10-25): the 4B's 2K prompt 911 tok/s against 242 (Vulkan) and 337 (SYCL); the 27B's 113 against 23 and 25. Flash Next's default is more accurate since 2026-11-19 (KL against f32 0.0153 -> 0.0046 on code) and a little slower: a 2K prompt 116 tok/s, generation 7.4.
Core Ultra 9 285H mini PC (Arc 140T)
Arc 140T (Xe-LPG Plus, 8-lane XMX) · 64 GB RAM, DDR5-5600 · Windows · PascalSoftwares, issue #5
| Model | Prompt 4K | Generation | With drafting | Release |
|---|---|---|---|---|
| Qwen3.8 27B | 405 | 6.1 | 14.6 | 2026.11.18.07 |
Generation reads its weights at ~84 % of what this machine's memory delivers, near its limit.
Core Ultra 7 258V (Arc 140V)
Arc 140V (Lunar Lake, Xe2, 16-lane XMX) · LPDDR5X-8533, 31 GB · Windows 11 · a1ex4, issue #60
| Model | Prompt 4K | Generation | With drafting | Release |
|---|---|---|---|---|
| Qwen3.8 27B | 401-444 | 5.9-6.5 | 13.9 | 2026.11.19.02 |
The prompt slows over three passes (444, 416, 401 tok/s) as its clocks drop. Generation reads 73-81 GB/s of the 104-108 GB/s its memory gives. KL against the f32 reference 0.00117 (top token 97.9 %). 2K prompt 389 tok/s, 32K 268. The NPU runs on Lunar Lake, but the GPU alone is faster (4K 324 against 175 tok/s with it).
Core Ultra X7 358H mini PC (Arc B390)
Arc B390 (Panther Lake, Xe3, 16-lane XMX) · a 64 GB+ GPU share · issue #1
| Model | Prompt 4K | Generation | With drafting | Release |
|---|---|---|---|---|
| Qwen3.8 27B | 637 | 5.9 | 16.8 | 2026.11.18.06 |
| Qwen3.6 35B-A3B | 1,370 | 26 | 46 | 2026.11.18.06 |
Arc A770 16 GB
Alchemist (Xe-HPG) · Windows 11 · hansong80 and TheUniKian, issues #35, #36, #45; Linux (Ubuntu, xe driver, PCIe 3.0 x16) · skipzx-lab, #45
| Model | Prompt 4K | Generation | With drafting | Release |
|---|---|---|---|---|
| Qwen3.5 4B | 4,821 | 37.6-38.3 | 63.9 | 2026.11.18.02 (generation .18.07) |
| Qwen3.5 9B | 2,969-3,017 | 29.7-30.4 | 51.5 | 2026.11.17.08 (generation .18.07) |
| Qwen3.8 27B | 1,223 | 21.6-22.7 | 30.1-31.3 | 2026.11.17.08 / .18.08 |
| Qwen3.5 4B, Linux | – | 80.7-82.1 | 105.7 | 2026.11.18.08 |
| Qwen3.5 9B, Linux | – | 51.2-51.6 | 84.4 | 2026.11.18.08 |
| Qwen3.8 27B, Linux (this card held to 150 W) | 1,217 (16K: 1,135) | 22.6-22.9 | 40.4 | main fb00781 (prompt); 2026.11.18.08 |
| Qwen3.8 Flash Next IQ3_S, Linux (one card, 14-16 % of its experts cached, 150 W) | 428 (16K: 535) | 13.3 (15.1 with 2,048-token prompt chunks) | – | main 566d84e (after a 16K prompt) |
| Qwen3.6 35B-A3B, Linux (73 % of its experts on the card) | – | 52.4-70.0 | 97.6 | main at fd858b5 (the #56 fix) |
Linux: prompts of 2K / 8K tokens, no 4K run: 4B 4,518 / 5,000 tok/s, 9B 2,794 / 3,249, 27B 1,131 / 1,205, 35B-A3B 1,830 / 2,966 (131K: 1,133, generation 35.4; its whole 262K context: 709 tok/s, then 32.3). Generation at a 128-token depth and a 256-token coding answer.
The 27B's generation is with kernels sent in recorded lists (ARC_BATCH=64), the default on Alchemist under Windows since 2026.11.18.08; 12.6-12.8 tok/s before. With it, the 27B's drafting no longer adds speed (20.4 tok/s against 21.4 plain on a 512-token prompt).
Arc B580 12 GB
Battlemage (Xe2) · PCIe 3.0 slot · Linux (NixOS) · d-dorok, issue #37
| Model | Prompt 4K | Generation | Release |
|---|---|---|---|
| Qwen3.5 4B | 7,915 | 92.6 | commit 7703141 |
| Qwen3.5 9B | 5,455 | 56.2 | commit 651efd8 |
| Qwen3.6 35B-A3B | 2,469 | 84.7 | commit 7703141 |
Qwen3.8 27B with ARC_HOST_KEEP_GB=0.75: 0.49 GiB of it in RAM, computed by the CPU (the default since 2026-11-19; 7.1 read over PCIe) | 1,877 | 13.7 | commit 438dfc4; generation 10e0707 with ARC_HOST_CPU=1 |
| Qwen3.8 27B default room: 0.85-1.02 GiB of it in RAM, computed by the CPU (3.5-4.5 tok/s read over PCIe before) | 1,783 | 10.3-11.9 | commit 992ea08 |
The 27B is a little larger than this card can hold beside its buffers, so part of it is read from RAM over PCIe for every token.
The same files with llama.cpp on this card
| Model | inferzero | llama.cpp SYCL | llama.cpp Vulkan |
|---|---|---|---|
| Qwen3.5 4B Q4_K_M | 7,915 / 92.6 | 1,520 / 95.4 | 2,411 / 79.7 |
| Qwen3.5 9B Q4_K_M | 5,455 / 56.2 | 864 / 61.1 | 1,703 / 53.0 |
| Qwen3.6 35B-A3B UD-IQ4_XS | 2,469 / 84.7 | 535 / 41.9 | 225 / 29.1 |
| Qwen3.8 27B IQ3_S | 1,877 / 13.7 (7.1 before the CPU took the RAM-kept weights) | 213 / 11.4 | 486 / 4.9-5.3 |
4K prompt / generation, tok/s; d-dorok, llama.cpp b1b770d (llama-bench -p 4096 -n 64 -fa 1, the experts or layers that do not fit on the CPU). llama.cpp is 3-9 % ahead on the small models' generation. On the 27B, inferzero now does what llama.cpp does with the layers that do not fit: the CPU computes the part kept in RAM (the default on discrete cards since 2026-11-19), and generation went from 7.1 to 13.7 tok/s, against llama.cpp SYCL's 11.4.
Arc Pro B50 16 GB
Battlemage (Xe2) · PCIe 3.0 x8 · Linux (Unraid, Docker) · catduckgnaf, issue #59
| Model | Prompt | Generation | Release |
|---|---|---|---|
| Ornith 1.5 35B-A3B Q4_K_M (a Qwen3.6 35B-A3B fine-tune) | ~1,670 | 40-46 | 2026.11.16.16 |
| Gemma 4 26B-A4B QAT Q4_0 | ~660 | 33-38 | 2026.11.16.16 |
serverz answers of 400 tokens, prompts of a chat's size.
Arc Pro B60 24 GB
Battlemage (Xe2) · Windows · manoreken2, issue #4; Linux (Ubuntu, xe driver, PCIe 3.0 x8) · skipzx-lab, issue #57
| Model | Prompt 4K | Generation | With drafting | Release |
|---|---|---|---|---|
| Qwen3.8 27B | 1,726 | 22.7-23.7 | 51.6 | 2026.11.17.08 (generation .14.09, drafting .13.11) |
| Qwen3.6 35B-A3B | 3,536 | 82.4 | 111.9 | 2026.11.14.09 (generation .13.11) |
| Qwen3.5 4B, Linux | – | 84.7-86.3 | 115.2 | 2026.11.18.08 |
| Qwen3.5 9B, Linux | – | 52.3-52.8 | 82.9 | 2026.11.18.08 |
Qwen3.8 27B, Linux the FFN's gate and up in one GEMM launch (ARC_GEMM_XW_GU, the default since 2026-11-20; 1,806 without it) | 1,874 (16K: 1,832) | 23.2-23.3 | 50.0 | main a5acc31 = release 2026.11.20.03 (prompt); 2026.11.18.08 |
| Qwen3.6 35B-A3B, Linux | – | 83.9-85.6 | 98.5 | 2026.11.18.08 |
| Qwen3.8 Flash Next IQ3_S, Linux (25-30 % of its experts on the card) | 531 (16K: 672, 32K: 629) | 22.3-22.7 (26.1-26.3 with ARC_XCPU=1; precise mode 18.6-21.0, 20.4-23.0 with it) | – | main 566d84e / a62be02 (prompt chunk 4,096) |
Linux: prompts of 2K / 8K tokens, no 4K run: 4B 7,174 / 7,007 tok/s, 9B 4,874 / 4,862, 27B 1,610 / 1,754, 35B-A3B 3,236 / 3,995; 128K: 4B 3,137, 27B 905, 35B-A3B 2,129. Flash Next's experts are streamed over the PCIe 3.0 x8 link; ARC_XCPU=1 (pull request #55) computes the experts missing from the GPU's cache on the CPU.
Arc Pro B65 32 GB
Battlemage (Xe2) · Linux (TUXEDO OS) · teodortomas, issue #47
| Model | Prompt 4K | Generation | With drafting | Release |
|---|---|---|---|---|
| Qwen3.5 4B UD-Q4_K_XL | 7,774 | 81.2 | 115.5 | 2026.11.18.08 |
| Qwen3.5 9B UD-Q4_K_XL | 5,226 | 49.2 | 79.0 | 2026.11.18.08 |
| Qwen3.6 35B-A3B UD-IQ4_XS | 3,512 | 82.0 | 99.8 | 2026.11.18.08 |
| Qwen3.8 27B | 1,735 | 21.6-22.0 | 46.7-51.8 | 2026.11.18.08 (drafting 51.8: .19.06, bench/decode_try.sh) |
At 131K tokens: the 4B's prompt 2,960 tok/s and generation 57.8, the 9B's 2,474 and 39.1, the 35B-A3B's 1,830 and 60.3.
Arc Pro B70 32 GB
Battlemage (Xe2) · Linux (Fedora 45) · joshdev-de, issue #27
| Model | Prompt | Generation | With drafting | Release |
|---|---|---|---|---|
| Qwen3.8 27B the FFN's gate and up in one GEMM launch, the default since 2026-11-20 (2,972 without it) | 3,100 4K 2,727 8K | 35.2 | 70.6 | main e93fac1 with ARC_GEMM_XW_GU=1, = release 2026.11.20.03 (8K .16.12, generation .16.02) |
| Qwen3.6 35B-A3B | 5,177 4K 5,647 8K 5,445 16K | 117.2 after a 4K prompt, 115.5 after 16K | – | 2026.11.17.04 (generation main e93fac1) |
| Qwen3.8 Flash Next 46 % of its experts on the card | 1,414 4K 1,145 16K | 44.9 warm, the CPU computing the missed experts (44.2 without) 6.6 cold | – | 2026.11.17.04; generation a5acc31 = release 2026.11.20.03 |
Another Pro B70 under Windows (issue #12, release 2026.11.13.06, before several prompt changes): the 27B's 4K prompt 2,105 tok/s, generation 32.2, 75.2 with drafting.
Intel Data Center GPU Max 1550
Ponte Vecchio (Xe-HPC), two stacks a card · Linux, driver 25.18 · hibagus, pull request #51, #48
| Model | Prompt 512 | Prompt 4K | Release |
|---|---|---|---|
| Qwen3.8 27B, one stack | 1,939 | 2,642 | main, 2026-11-19 |
Qwen3.8 27B, both stacks (--gpus 2) | – | 3,380 | main, 2026-11-19 |
Qwen3.8 27B, 4 stacks on 2 cards (--gpus 4, Xe Link) | 2,676-2,697 | – | main at e2ae1d2 (2026.11.19.07's default) |
2, 3 and 4 × Arc Pro B60: one model split
One model's layers divided over the cards (--gpus N) · Windows (no direct GPU-to-GPU copies there: the sums go through host memory) · manoreken2, issue #4
| Model | 1 card | 2 cards | 3 cards | 4 cards |
|---|---|---|---|---|
| Qwen3.8 27B, prompt 4K | 1,726 | 1,597 | 1,724 | 1,888 |
| Qwen3.8 27B, generation | 22.7 | 35.7 | 43 | 53 |
| DeepSeek-V4.1-Flash IQ2_XS, generation | 10.3 | 18.0 | 16.6 | 17.6 |
Releases: the 27B's split prompts 2026.11.13.05 / .11 and its split generation a run of 2026-10-31, both older than the one-card figures; DeepSeek 2026.11.14.09 (machine with 1 TB of RAM). Long prompts over 4 cards (2026.11.14.05): 64K / 131K / 262K tokens at 2,152 / 1,267 / 700 tok/s for the 35B-A3B, 1,258 / 970 / 640 for the 27B and 948 / 778 / 526 for Flash Next; the 35B-A3B's 4K prompt 4,264 tok/s.
Several conversations at once
Batched decoding (serverz --slots: several chats' next tokens in one pass), runz --bench-batch 4 · Linux, one card at a time · skipzx-lab, issue #58, main at fd858b5
| Model | Pro B60: 4 together / 1 alone | A770: 4 together / 1 alone |
|---|---|---|
| Qwen3.5 4B | 235.2 / 90.9 (2.59x) | 203.2 / 86.3 (2.35x) |
| Qwen3.8 27B | 70.2 / 23.6 (2.97x) | 62.4 / 22.8 (2.74x) |
| Qwen3.6 35B-A3B | 182.3 / 89.7 (2.03x) | 118.2 / 67.4 (1.76x) (73 % of its experts on the card) |
| Qwen3.8 Flash Next IQ3_S | 52.2 / 28.8 (1.81x) (30 % of its experts on the card) | – |
Tokens per second, all conversations together against one alone. Each conversation's tokens are the same, bit for bit, as when it runs alone. The two cards' runs overlapped in time, so the speeds may read a little low.
One model split over 2 Pro B60s (--gpus 2, PCIe x16 + x4, the sums through host memory) · MaineTim, issue #58, main at f3174f2
| Model | 1 card: 4 together / 1 alone | 2 cards: 4 together / 1 alone |
|---|---|---|
| Qwen3.8 27B | 70.1 / 23.7 (2.96x) | 92.9 / 37.5 (2.48x) |
| Qwen3.6 35B-A3B Q4_K_M | 159.3 / 81.7 (1.95x) | 202.5 / 104.9 (1.96x) |
| Qwen3.8 Flash Next IQ3_XXS | 40.3 / 23.3 (1.72x) (37 % of its experts on the card) | 68.3 / 32.8 (2.08x) |
Split over both cards, too, every conversation's tokens are bit for bit those it gets alone.
Two Arc A770s, Windows · hansong80, release 2026.11.19.06 · and one Pro B65 32 GB, Linux · teodortomas, release 2026.11.19.06 · issue #58
| Model | A770: 4 together / 1 alone | 2 × A770: 4 together / 1 alone | Pro B65: 4 together / 1 alone |
|---|---|---|---|
| Qwen3.5 4B | 189.1 / 74.9 (2.54x) | 144.2 / 41.5 (3.54x) | 225.8 / 84.5 (2.65x) |
| Qwen3.8 27B | 58.6 / 21.2 (2.77x) | 52.1 / 16.2 (3.21x) | 65.1 / 21.5 (3.03x) |
| Qwen3.6 35B-A3B | 66.3 / 20.4 (3.25x) (76 % of its experts on the card) | 89.0 / 28.3 (3.18x) (every expert on the cards) | 172.8 / 90.4 (2.01x) |
A Pro B70 (joshdev-de, Linux, main 566d84e), 4 together / 1 alone: the 27B 107.3 / 36.4 (2.95x), the 35B-A3B 262.1 / 132.6 (2.03x), Flash Next 105.3 / 39.1, all bit-identical.
Every run bit-identical. On the B65 Flash Next IQ3_S too: 65.0 tok/s for four, 32.3 alone (45 % of its experts on the card). On the two A770s the 35B-A3B is faster split than on one card (28.3 against 20.4 tok/s for one conversation): each card then holds all of its half's experts, so none comes from the SSD; the 4B and the 27B are slower split there (the sums go through host memory under Windows), and batching is what makes the split pay.
Other multi-GPU setups
| Setup | Prompt 4K | Generation | Notes |
|---|---|---|---|
| 2 × Arc Pro B60, Linux, PCIe 4.0 x8 each scenic6942, #17 | 1,630 | 37.8 | the sums straight from GPU to GPU (1,442 through host memory) |
| 2 × Arc Pro B60, PCIe x16 + x4 MaineTim, #58 | – | 37.5 | one conversation; the 35B-A3B 104.9 tok/s (81.7 on one card), 4 conversations together 202.5 |
| 2 × Arc Pro B50, Linux, PCIe 3.0 x8 each, no peer access catduckgnaf, #59 | 975 (16.5K) | 19.3-19.6 | the 27B Heretic Q4_K_M; llama.cpp SYCL's layer split ~290 / ~11. A MoE that fits one card is faster unsplit here (35B-A3B 40-46 alone, 8.2 split) |
| 2 × Arc A770, Windows, Flash Next IQ3_S hansong80, #45 | 528 | 13.9 | Qwen3.8 Flash Next GSQ-RCO IQ3_S (the file the other cards ran), release 2026.11.20.02, 4K prompt; 36 % of each card's half of the experts cached (RAM tier 16 GB), 96 sums a token (11 ms of 71); ARC_HOST_KEEP_GB=60 no gain here |
| 2 × Arc A770, Windows, Flash Next IQ2_XS hansong80, #45 | 657 | 15.1-15.9 | Qwen3.8 Flash Next (GSQ-RCO abliterated IQ2_XS, 68.5 GB), release 2026.11.19.06; 58 % of each card's half of the experts cached; the first answer after loading 6.8 while the cache fills |
| 2 × Arc A770, Windows hansong80, #35 | 1,126 | 12.2 | no faster than one card yet: every sum waits on a host round trip |
| 2 × Arc Pro B60, one in a PCIe x4 slot MaineTim, #19 | 620 | – | the 27B UD-Q5_K_XL; one card alone 1,223: a slow link halves a split model's prompt |