inferzero github · 2026.11.19.01

Measured results, setup by setup

Every number below was measured on a real machine, most of them by the people who own it, and posted on GitHub with its log. Each table is one setup: its GPU (or GPUs), the system it ran on, and the models tried there.

Prompt
prompt processing (pp): how fast a prompt is read, in tokens a second, at the prompt length shown.
Generation
token generation (tg): how fast the answer is written, in tokens a second; one token is about three quarters of a word.
With drafting
generation with the model's draft head (MTP, DFlash2 or Gemma's drafter), which gives exactly the same text.

Models are Qwen3.8 27B GSQ-RCO IQ3_S unless named; the others are the files the launcher downloads. Numbers move with each release, so every row says which one. Setups we have not measured are the most useful thing you can add: how to measure yours.

Core Ultra 5 125H laptop

Arc graphics (Xe-LPG, 8 Xe-cores, DP4a, no XMX) · 32 GB RAM · Windows 11 · the laptop inferzero was built on · bench/website_bench.ps1, 2026-11-09

ModelPrompt 8K32K131KGenerationWith drafting
Qwen3.5 4B89670536621.035.8
Gemma 4 E4B70749422219.141.8
Qwen3.5 9B48743527811.821.6
Qwen3.6 35B-A3B experts on the SSD85163732421.638.1
Gemma 4 26B-A4B experts on the SSD2321244419.8–
Qwen3.8 27B135125815.112.9
Qwen3.8 Flash Next 52 GB of experts on the SSD2051741466.37.6

Generation over 256 tokens. Against llama.cpp on the same laptop and files (2026-10-25): the 4B's 2K prompt 911 tok/s against 242 (Vulkan) and 337 (SYCL); the 27B's 113 against 23 and 25. Flash Next's default is more accurate since 2026-11-19 (KL against f32 0.0153 -> 0.0046 on code) and a little slower: a 2K prompt 116 tok/s, generation 7.4.

Core Ultra 9 285H mini PC (Arc 140T)

Arc 140T (Xe-LPG Plus, 8-lane XMX) · 64 GB RAM, DDR5-5600 · Windows · PascalSoftwares, issue #5

ModelPrompt 4KGenerationWith draftingRelease
Qwen3.8 27B4056.114.62026.11.18.07

Generation reads its weights at ~84 % of what this machine's memory delivers, near its limit.

Core Ultra 7 258V (Arc 140V)

Arc 140V (Lunar Lake, Xe2, 16-lane XMX) · LPDDR5X-8533, 31 GB · Windows 11 · a1ex4, issue #60

ModelPrompt 4KGenerationWith draftingRelease
Qwen3.8 27B401-4445.9-6.513.92026.11.19.02

The prompt slows over three passes (444, 416, 401 tok/s) as its clocks drop. Generation reads 73-81 GB/s of the 104-108 GB/s its memory gives. KL against the f32 reference 0.00117 (top token 97.9 %). 2K prompt 389 tok/s, 32K 268. The NPU runs on Lunar Lake, but the GPU alone is faster (4K 324 against 175 tok/s with it).

Core Ultra X7 358H mini PC (Arc B390)

Arc B390 (Panther Lake, Xe3, 16-lane XMX) · a 64 GB+ GPU share · issue #1

ModelPrompt 4KGenerationWith draftingRelease
Qwen3.8 27B6375.916.82026.11.18.06
Qwen3.6 35B-A3B1,37026462026.11.18.06

Arc A770 16 GB

Alchemist (Xe-HPG) · Windows 11 · hansong80 and TheUniKian, issues #35, #36, #45; Linux (Ubuntu, xe driver, PCIe 3.0 x16) · skipzx-lab, #45

ModelPrompt 4KGenerationWith draftingRelease
Qwen3.5 4B4,82137.6-38.363.92026.11.18.02 (generation .18.07)
Qwen3.5 9B2,969-3,01729.7-30.451.52026.11.17.08 (generation .18.07)
Qwen3.8 27B1,22321.6-22.730.1-31.32026.11.17.08 / .18.08
Qwen3.5 4B, Linux–80.7-82.1105.72026.11.18.08
Qwen3.5 9B, Linux–51.2-51.684.42026.11.18.08
Qwen3.8 27B, Linux (this card held to 150 W)1,217 (16K: 1,135)22.6-22.940.4main fb00781 (prompt); 2026.11.18.08
Qwen3.8 Flash Next IQ3_S, Linux (one card, 14-16 % of its experts cached, 150 W)428 (16K: 535)13.3 (15.1 with 2,048-token prompt chunks)–main 566d84e (after a 16K prompt)
Qwen3.6 35B-A3B, Linux (73 % of its experts on the card)–52.4-70.097.6main at fd858b5 (the #56 fix)

Linux: prompts of 2K / 8K tokens, no 4K run: 4B 4,518 / 5,000 tok/s, 9B 2,794 / 3,249, 27B 1,131 / 1,205, 35B-A3B 1,830 / 2,966 (131K: 1,133, generation 35.4; its whole 262K context: 709 tok/s, then 32.3). Generation at a 128-token depth and a 256-token coding answer.

The 27B's generation is with kernels sent in recorded lists (ARC_BATCH=64), the default on Alchemist under Windows since 2026.11.18.08; 12.6-12.8 tok/s before. With it, the 27B's drafting no longer adds speed (20.4 tok/s against 21.4 plain on a 512-token prompt).

Arc B580 12 GB

Battlemage (Xe2) · PCIe 3.0 slot · Linux (NixOS) · d-dorok, issue #37

ModelPrompt 4KGenerationRelease
Qwen3.5 4B7,91592.6commit 7703141
Qwen3.5 9B5,45556.2commit 651efd8
Qwen3.6 35B-A3B2,46984.7commit 7703141
Qwen3.8 27B with ARC_HOST_KEEP_GB=0.75: 0.49 GiB of it in RAM, computed by the CPU (the default since 2026-11-19; 7.1 read over PCIe)1,87713.7commit 438dfc4; generation 10e0707 with ARC_HOST_CPU=1
Qwen3.8 27B default room: 0.85-1.02 GiB of it in RAM, computed by the CPU (3.5-4.5 tok/s read over PCIe before)1,78310.3-11.9commit 992ea08

The 27B is a little larger than this card can hold beside its buffers, so part of it is read from RAM over PCIe for every token.

The same files with llama.cpp on this card

Modelinferzerollama.cpp SYCLllama.cpp Vulkan
Qwen3.5 4B Q4_K_M7,915 / 92.61,520 / 95.42,411 / 79.7
Qwen3.5 9B Q4_K_M5,455 / 56.2864 / 61.11,703 / 53.0
Qwen3.6 35B-A3B UD-IQ4_XS2,469 / 84.7535 / 41.9225 / 29.1
Qwen3.8 27B IQ3_S1,877 / 13.7 (7.1 before the CPU took the RAM-kept weights)213 / 11.4486 / 4.9-5.3

4K prompt / generation, tok/s; d-dorok, llama.cpp b1b770d (llama-bench -p 4096 -n 64 -fa 1, the experts or layers that do not fit on the CPU). llama.cpp is 3-9 % ahead on the small models' generation. On the 27B, inferzero now does what llama.cpp does with the layers that do not fit: the CPU computes the part kept in RAM (the default on discrete cards since 2026-11-19), and generation went from 7.1 to 13.7 tok/s, against llama.cpp SYCL's 11.4.

Arc Pro B50 16 GB

Battlemage (Xe2) · PCIe 3.0 x8 · Linux (Unraid, Docker) · catduckgnaf, issue #59

ModelPromptGenerationRelease
Ornith 1.5 35B-A3B Q4_K_M (a Qwen3.6 35B-A3B fine-tune)~1,67040-462026.11.16.16
Gemma 4 26B-A4B QAT Q4_0~66033-382026.11.16.16

serverz answers of 400 tokens, prompts of a chat's size.

Arc Pro B60 24 GB

Battlemage (Xe2) · Windows · manoreken2, issue #4; Linux (Ubuntu, xe driver, PCIe 3.0 x8) · skipzx-lab, issue #57

ModelPrompt 4KGenerationWith draftingRelease
Qwen3.8 27B1,72622.7-23.751.62026.11.17.08 (generation .14.09, drafting .13.11)
Qwen3.6 35B-A3B3,53682.4111.92026.11.14.09 (generation .13.11)
Qwen3.5 4B, Linux–84.7-86.3115.22026.11.18.08
Qwen3.5 9B, Linux–52.3-52.882.92026.11.18.08
Qwen3.8 27B, Linux the FFN's gate and up in one GEMM launch (ARC_GEMM_XW_GU, the default since 2026-11-20; 1,806 without it)1,874 (16K: 1,832)23.2-23.350.0main a5acc31 = release 2026.11.20.03 (prompt); 2026.11.18.08
Qwen3.6 35B-A3B, Linux–83.9-85.698.52026.11.18.08
Qwen3.8 Flash Next IQ3_S, Linux (25-30 % of its experts on the card)531 (16K: 672, 32K: 629)22.3-22.7 (26.1-26.3 with ARC_XCPU=1; precise mode 18.6-21.0, 20.4-23.0 with it)–main 566d84e / a62be02 (prompt chunk 4,096)

Linux: prompts of 2K / 8K tokens, no 4K run: 4B 7,174 / 7,007 tok/s, 9B 4,874 / 4,862, 27B 1,610 / 1,754, 35B-A3B 3,236 / 3,995; 128K: 4B 3,137, 27B 905, 35B-A3B 2,129. Flash Next's experts are streamed over the PCIe 3.0 x8 link; ARC_XCPU=1 (pull request #55) computes the experts missing from the GPU's cache on the CPU.

Arc Pro B65 32 GB

Battlemage (Xe2) · Linux (TUXEDO OS) · teodortomas, issue #47

ModelPrompt 4KGenerationWith draftingRelease
Qwen3.5 4B UD-Q4_K_XL7,77481.2115.52026.11.18.08
Qwen3.5 9B UD-Q4_K_XL5,22649.279.02026.11.18.08
Qwen3.6 35B-A3B UD-IQ4_XS3,51282.099.82026.11.18.08
Qwen3.8 27B1,73521.6-22.046.7-51.82026.11.18.08 (drafting 51.8: .19.06, bench/decode_try.sh)

At 131K tokens: the 4B's prompt 2,960 tok/s and generation 57.8, the 9B's 2,474 and 39.1, the 35B-A3B's 1,830 and 60.3.

Arc Pro B70 32 GB

Battlemage (Xe2) · Linux (Fedora 45) · joshdev-de, issue #27

ModelPromptGenerationWith draftingRelease
Qwen3.8 27B the FFN's gate and up in one GEMM launch, the default since 2026-11-20 (2,972 without it)3,100 4K
2,727 8K
35.270.6main e93fac1 with ARC_GEMM_XW_GU=1, = release 2026.11.20.03 (8K .16.12, generation .16.02)
Qwen3.6 35B-A3B5,177 4K
5,647 8K
5,445 16K
117.2 after a 4K prompt, 115.5 after 16K–2026.11.17.04 (generation main e93fac1)
Qwen3.8 Flash Next 46 % of its experts on the card1,414 4K
1,145 16K
44.9 warm, the CPU computing the missed experts (44.2 without)
6.6 cold
–2026.11.17.04; generation a5acc31 = release 2026.11.20.03

Another Pro B70 under Windows (issue #12, release 2026.11.13.06, before several prompt changes): the 27B's 4K prompt 2,105 tok/s, generation 32.2, 75.2 with drafting.

Intel Data Center GPU Max 1550

Ponte Vecchio (Xe-HPC), two stacks a card · Linux, driver 25.18 · hibagus, pull request #51, #48

ModelPrompt 512Prompt 4KRelease
Qwen3.8 27B, one stack1,9392,642main, 2026-11-19
Qwen3.8 27B, both stacks (--gpus 2)–3,380main, 2026-11-19
Qwen3.8 27B, 4 stacks on 2 cards (--gpus 4, Xe Link)2,676-2,697–main at e2ae1d2 (2026.11.19.07's default)

2, 3 and 4 × Arc Pro B60: one model split

One model's layers divided over the cards (--gpus N) · Windows (no direct GPU-to-GPU copies there: the sums go through host memory) · manoreken2, issue #4

Model1 card2 cards3 cards4 cards
Qwen3.8 27B, prompt 4K1,7261,5971,7241,888
Qwen3.8 27B, generation22.735.74353
DeepSeek-V4.1-Flash IQ2_XS, generation10.318.016.617.6

Releases: the 27B's split prompts 2026.11.13.05 / .11 and its split generation a run of 2026-10-31, both older than the one-card figures; DeepSeek 2026.11.14.09 (machine with 1 TB of RAM). Long prompts over 4 cards (2026.11.14.05): 64K / 131K / 262K tokens at 2,152 / 1,267 / 700 tok/s for the 35B-A3B, 1,258 / 970 / 640 for the 27B and 948 / 778 / 526 for Flash Next; the 35B-A3B's 4K prompt 4,264 tok/s.

Several conversations at once

Batched decoding (serverz --slots: several chats' next tokens in one pass), runz --bench-batch 4 · Linux, one card at a time · skipzx-lab, issue #58, main at fd858b5

ModelPro B60: 4 together / 1 aloneA770: 4 together / 1 alone
Qwen3.5 4B235.2 / 90.9 (2.59x)203.2 / 86.3 (2.35x)
Qwen3.8 27B70.2 / 23.6 (2.97x)62.4 / 22.8 (2.74x)
Qwen3.6 35B-A3B182.3 / 89.7 (2.03x)118.2 / 67.4 (1.76x) (73 % of its experts on the card)
Qwen3.8 Flash Next IQ3_S52.2 / 28.8 (1.81x) (30 % of its experts on the card)–

Tokens per second, all conversations together against one alone. Each conversation's tokens are the same, bit for bit, as when it runs alone. The two cards' runs overlapped in time, so the speeds may read a little low.

One model split over 2 Pro B60s (--gpus 2, PCIe x16 + x4, the sums through host memory) · MaineTim, issue #58, main at f3174f2

Model1 card: 4 together / 1 alone2 cards: 4 together / 1 alone
Qwen3.8 27B70.1 / 23.7 (2.96x)92.9 / 37.5 (2.48x)
Qwen3.6 35B-A3B Q4_K_M159.3 / 81.7 (1.95x)202.5 / 104.9 (1.96x)
Qwen3.8 Flash Next IQ3_XXS40.3 / 23.3 (1.72x) (37 % of its experts on the card)68.3 / 32.8 (2.08x)

Split over both cards, too, every conversation's tokens are bit for bit those it gets alone.

Two Arc A770s, Windows · hansong80, release 2026.11.19.06 · and one Pro B65 32 GB, Linux · teodortomas, release 2026.11.19.06 · issue #58

ModelA770: 4 together / 1 alone2 × A770: 4 together / 1 alonePro B65: 4 together / 1 alone
Qwen3.5 4B189.1 / 74.9 (2.54x)144.2 / 41.5 (3.54x)225.8 / 84.5 (2.65x)
Qwen3.8 27B58.6 / 21.2 (2.77x)52.1 / 16.2 (3.21x)65.1 / 21.5 (3.03x)
Qwen3.6 35B-A3B66.3 / 20.4 (3.25x) (76 % of its experts on the card)89.0 / 28.3 (3.18x) (every expert on the cards)172.8 / 90.4 (2.01x)

A Pro B70 (joshdev-de, Linux, main 566d84e), 4 together / 1 alone: the 27B 107.3 / 36.4 (2.95x), the 35B-A3B 262.1 / 132.6 (2.03x), Flash Next 105.3 / 39.1, all bit-identical.

Every run bit-identical. On the B65 Flash Next IQ3_S too: 65.0 tok/s for four, 32.3 alone (45 % of its experts on the card). On the two A770s the 35B-A3B is faster split than on one card (28.3 against 20.4 tok/s for one conversation): each card then holds all of its half's experts, so none comes from the SSD; the 4B and the 27B are slower split there (the sums go through host memory under Windows), and batching is what makes the split pay.

Other multi-GPU setups

SetupPrompt 4KGenerationNotes
2 × Arc Pro B60, Linux, PCIe 4.0 x8 each scenic6942, #171,63037.8the sums straight from GPU to GPU (1,442 through host memory)
2 × Arc Pro B60, PCIe x16 + x4 MaineTim, #58–37.5one conversation; the 35B-A3B 104.9 tok/s (81.7 on one card), 4 conversations together 202.5
2 × Arc Pro B50, Linux, PCIe 3.0 x8 each, no peer access catduckgnaf, #59975 (16.5K)19.3-19.6the 27B Heretic Q4_K_M; llama.cpp SYCL's layer split ~290 / ~11. A MoE that fits one card is faster unsplit here (35B-A3B 40-46 alone, 8.2 split)
2 × Arc A770, Windows, Flash Next IQ3_S hansong80, #4552813.9Qwen3.8 Flash Next GSQ-RCO IQ3_S (the file the other cards ran), release 2026.11.20.02, 4K prompt; 36 % of each card's half of the experts cached (RAM tier 16 GB), 96 sums a token (11 ms of 71); ARC_HOST_KEEP_GB=60 no gain here
2 × Arc A770, Windows, Flash Next IQ2_XS hansong80, #4565715.1-15.9Qwen3.8 Flash Next (GSQ-RCO abliterated IQ2_XS, 68.5 GB), release 2026.11.19.06; 58 % of each card's half of the experts cached; the first answer after loading 6.8 while the cache fills
2 × Arc A770, Windows hansong80, #351,12612.2no faster than one card yet: every sum waits on a host round trip
2 × Arc Pro B60, one in a PCIe x4 slot MaineTim, #19620–the 27B UD-Q5_K_XL; one card alone 1,223: a slow link halves a split model's prompt