Hardware
One engine, from a laptop iGPU to four Pro B60s. The same Qwen3.8 27B file on every machine.
The engine was built and tuned on a Core Ultra 5 125H, which has no matrix engines. Where a GPU has XMX, the prompt's matrix products move onto it after a start-up check against the DP4a kernels. One model can be split over several cards with --gpus N; where the driver offers peer access (Linux) the sums go straight from GPU to GPU. Discrete cards are experimental, their defaults set from their owners' runs.
Prompt · tok/s4K prompt unless noted
Core Ultra 5 125HXe-LPG · DP4a · 8K135
Core Ultra 9 285HArc 140T · 2026.11.18.07 defaults405
Core Ultra X7 358HArc B390 · 2026.11.18.06637
1× Arc Pro B60Windows · 2026.11.17.081,726
1× Arc B58012 GB · PCIe 3.0 · Linux · ARC_HOST_KEEP_GB=0.751,877
1× Arc Pro B6532 GB · Linux · 2026.11.18.081,735
1× Arc A770Alchemist · 2026.11.18.02 defaults1,223
1× Arc Pro B70Linux · main 7368768, ARC_H8_NORM2,972
2× Arc Pro B60Windows · 2026.11.13.051,597
3× Arc Pro B60Windows · 2026.11.13.111,724
4× Arc Pro B60Windows · 2026.11.13.051,888
Generation · tok/splain decoding
Core Ultra 5 125H12.9 with MTP + DFlash25.1
Core Ultra 9 285H16.9 with MTP + DFlash26.0
1× Arc B58012 GB · 0.49 GiB of weights in RAM over PCIe 3.07.1
1× Arc Pro B6546.7 with MTP · Linux · 2026.11.18.0821.6
1× Arc A77031.3 with MTP · ARC_BATCH=64, the default since 2026.11.18.0822.7
1× Arc Pro B602026.11.14.0922.7
1× Arc Pro B7070.6 with MTP · Linux · 2026.11.16.0235.2
2× Arc Pro B60Linux, GPU to GPU · 11-1437.8
3× Arc Pro B60Windows · run of 2026-10-3143
4× Arc Pro B60Windows · run of 2026-10-3153
the reference laptopmeasured by usersa shorter prompt, not comparable
Qwen3.8 27B GSQ-RCO IQ3_S; the laptop's prompt is 8K, the others 4K. Each bar is one user's run, with its release. The B70's prompt is a Pro B70 on native Linux (
issue #27, main 7368768 with
ARC_H8_NORM=1, the norm and the H8 rotation in one kernel, the default on discrete cards since 2026-11-18 (2,890 without it); with the flash attention of PR #29, the decoded weights kept resident, the H8 GEMM in 2D messages with its prefetch; 2,105 tok/s on another B70 under Windows before the weight prefetch,
issue #12); its generation bar is the same Linux B70 with the GEMV rows of joshdev-de's branch (the MTP figure through serverz, 4 drafts). The B580 (
issue #37, commit 7703141, PCIe 3.0) cannot hold all of the 27B's weights beside its buffers: with
ARC_HOST_KEEP_GB=0.75 0.49 GiB of them stay in RAM (1.94 tok/s generation with the old 2.24 GiB there). The several-B60 bars (
issue #4) are from before the weight prefetch that took one B60 from 1,322 to 1,522 tok/s, and newer runs are coming.
- One Pro B70 (issue #27, Linux), long prompts with main 7368768: the 27B 64K at 2,131 tok/s, 131K 1,614 (attention 31 / 48 % of the time), generation after them 30.1 / 26.3 tok/s; the 35B-A3B 64K 4,337. Qwen3.6 35B-A3B: an 8K prompt at 5,647 tok/s (2026.11.17.04), 4K 5,177 and 16K 5,445 (2026.11.17.03 / .04); generation 95.3-97.6 tok/s once its kernels are cached (2026.11.16.15). Qwen3.8 Flash Next on the same card, 46 % of its 52 GB of experts in the card's memory: 4K prompt 1,414 (2026.11.17.04, the expert gate / up through the XMX GEMM), 16K 1,145 (before it); generation 38.6 tok/s over 256 tokens after a README prompt, when the expert cache holds what the text uses (6.6 in the 64-token benchmark, which starts with it cold; the rest of the experts come from RAM).
- One Arc B580 12 GB (issue #37, Linux, PCIe 3.0, commit 7703141), 4K prompts: Qwen3.5 4B 7,915 tok/s and generation 92.6, Qwen3.5 9B 5,341 and 56.5, Qwen3.6 35B-A3B 2,469 and 84.7. The 27B by default (commit 438dfc4, 1.02 GiB of its weights in RAM): 1,783 and 3.53 tok/s.
- One Arc A770 16 GB (issue #35, Windows): Qwen3.5 4B's 4K prompt 4,821 tok/s through the Arc 140T preset (release 2026.11.18.02, the default since 2026.11.18.03), Qwen3.5 9B 2,969-3,017 (2026.11.17.08). Two A770s, no peer access: the 27B's prompt 1,126 against one card's 1,169 and generation 12.2 against 12.6-12.8, the sums' host round trips 73.5 ms of an 81 ms token (
ARC_TP_EVSUM=1, opt-in since 2026.11.18.05, is for that).
- One B60 with MTP: the 27B generates 51.6 tok/s, the 35B 82.4 and 111.9 (release 2026.11.13.11). The 35B's 4K prompt: 3,536 tok/s (2026.11.14.09).
- On Windows the B60s have no peer access, so several cards add their sums through host memory. On Linux two B60s (issue #17, PCIe 4.0 x8 each) sum straight from GPU to GPU: a 4K prompt at 1,630 tok/s against 1,442 through host memory, generation 37.8 against 36.1, the same logits.
- A card on a slow link sets the pace: one in an x4 slot (2.4 GB/s, issue #19, another 27B file) makes two cards half as fast as one on the prompt (620 against 1,223 tok/s).
- Qwen3.8 Flash Next (52 GB of experts) over 1 / 2 / 4 B60s: a 4K prompt at 665 / 1,202 / 1,491 tok/s, generation 5.7 / 34.4 / 32.7 (2026-11-13).
- DeepSeek-V4.1-Flash (ISTA's IQ2_XS) next to 1 TB of RAM: generation 10.3 / 18.0 / 16.6 / 17.6 tok/s over 1 / 2 / 3 / 4 B60s (release 2026.11.14.09). Unsloth's UD-Q8_K_XL over 4 B60s with 32 GiB of its experts per card kept in RAM (the default since 2026.11.16.16): a 512-token prompt at 24.0 tok/s, generation 9.25 (3.55 without).
- Panther Lake's Arc B390 (issue #1, Core Ultra X7 358H, 128 GB of DDR5-5600): the 27B's 4K prompt 258-268 tok/s on the plain XMX path, 546-560 with H8 on the wide-tile GEMM (the default since 2026.11.17.01), KL 0.0018 between the two, 583-592 with 16 GB of decoded weights resident as well (the default since 2026.11.17.02), 634-637 on release 2026.11.18.06; the 35B's 4K prompt 1,370 and generation 26 tok/s, 46 with MTP.
Details:
docs/multi-gpu.md,
docs/xe2-tuning.md. Have a card?
bench\website_bench.ps1 and
bench\decode_try.ps1 print everything needed; post the output on
issue #4.
| 4× Arc Pro B60, long prompts | 64K prompt | 131K prompt | 262K prompt | Generation after 131K | after 262K |
| Qwen3.6 35B-A3B MoE | 2,15230 s | 1,2671.7 min | 7006.2 min | 44.5 | 37.0 |
| Qwen3.8 27B | 1,25852 s | 9702.3 min | 6406.8 min | 32.2 | 29.0 |
| Qwen3.8 Flash Next MoE | 94869 s | 7782.8 min | 5268.3 min | 26.9 | 22.4 |
One run per model and length on a user's four Arc Pro B60s (issue #4, Windows, sums through host memory), release 2026.11.14.05, 2026-11-14: runz --gpus 4 --bench-pp N --n 32. Prompt in tok/s, the middle of three passes, with the time it took; the passes spread most for the 35B at 64K (1,857-2,187, the first pass the slowest) and 131K (1,230-1,360), and for the 27B at 262K (634-662). Generation is 32 tokens right after the prompt. Files: 35B UD-IQ4_XS, 27B and Flash Next GSQ-RCO IQ3_S.