● Level Zero · Intel Arc · laptop iGPU to Pro B60

A 35B model at 851 tok/s on the Intel laptop you already own. No discrete GPU, no Python.

An inference engine written from scratch for Intel Arc graphics. Qwen3.6 35B, Qwen3.8 27B and Flash Next's 52 GB of experts run on a Core Ultra laptop with 32 GB, and the same engine scales to four Arc Pro B60s. Every model is checked against an f32 reference, and the same prompt gives the same logits, bit for bit.

Windows 11 · Core Ultra with Arc graphics · 16 GB RAM for 4B / E4B, 32 GB for 12B-35B
Linux: the headless server and a terminal launcher, linux-x64 package · how to start · Arc B580 / Pro B60 / Pro B70 and A770 / A750 / A580: experimental · Apache-2.0

Core Ultra 5 125H
851tok/s

Qwen3.6 35B-A3B, 8K prompt, experts streamed from the SSD. A 131K-token context read in 6.7 minutes.

Core Ultra X7 358H
~637tok/s

Qwen3.8 27B, 4K prompt on Panther Lake's Arc B390, with the discrete cards' XMX path and 16 GB of decoded weights kept resident (the default since 2026.11.17.02; release 2026.11.18.06). The 285H's Arc 140T: 405.

1× Arc Pro B70
5,647tok/s

Qwen3.6 35B-A3B, 8K prompt; it generates 95-98 tok/s. The 27B: 2,897 tok/s at 4K, 2,727 at 8K. One B60: 3,536 and 1,726. A Pro B65: 1,735. An Arc A770: 1,223.

4× Arc Pro B60
4,264tok/s · 4K2,152tok/s · 64K

Qwen3.6 35B-A3B, one model split over four cards: a 4K and a 64K-token prompt. 1,267 at 131K, 700 at 262K.

Laptop: bench/website_bench.ps1, commit 281b083, 2026-11-09. B390: a user on issue #1, release 2026.11.18.06 with its defaults (613-637 tok/s over four sets of three runs; 583-592 on 2026.11.16.07, 546-560 without the resident weights). 140T: issue #5, release 2026.11.18.07 with its defaults (390 on 2026.11.17.07 with ARC_GEMM_XW_WGX=2). B70: a user on issue #27, native Linux: the 35B's 8K prompt on release 2026.11.17.04, its generation on 2026.11.16.15, the 27B's 4K best of 3 on main 1a7ac94 with the 2D GEMM and its prefetch (the defaults since 2026.11.17.06) and 8K on 2026.11.16.12. B580: issue #37, main 7703141, Linux, with ARC_HOST_KEEP_GB=0.75: 0.49 GiB of the 27B's weights in RAM on its 12 GB. A770: issue #36, release 2026.11.17.08 with ARC_GEMM_SPLIT=17, the default on Alchemist since 2026.11.18.02 (the Arc 140T preset); its generation on issue #45, release 2026.11.18.07 with ARC_BATCH=64, the default on Alchemist on Windows since 2026.11.18.08 (12.8 before). B65: issue #47, release 2026.11.18.08, Linux (bench/website_bench.sh). B60s: issue #4, release 2026.11.17.08 (one card, the 4K prompt; the 35B on 2026.11.14.09); four cards: the 4K prompt on 2026-11-02, the long prompts on release 2026.11.14.05.

Hardware

One engine, from a laptop iGPU to four Pro B60s. The same Qwen3.8 27B file on every machine.

The engine was built and tuned on a Core Ultra 5 125H, which has no matrix engines. Where a GPU has XMX, the prompt's matrix products move onto it after a start-up check against the DP4a kernels. One model can be split over several cards with --gpus N; where the driver offers peer access (Linux) the sums go straight from GPU to GPU. Discrete cards are experimental, their defaults set from their owners' runs.

Prompt · tok/s4K prompt unless noted
Core Ultra 5 125HXe-LPG · DP4a · 8K135
Core Ultra 9 285HArc 140T · 2026.11.18.07 defaults405
Core Ultra X7 358HArc B390 · 2026.11.18.06637
1× Arc Pro B60Windows · 2026.11.17.081,726
1× Arc B58012 GB · PCIe 3.0 · Linux · ARC_HOST_KEEP_GB=0.751,877
1× Arc Pro B6532 GB · Linux · 2026.11.18.081,735
1× Arc A770Alchemist · 2026.11.18.02 defaults1,223
1× Arc Pro B70Linux · main 7368768, ARC_H8_NORM2,972
2× Arc Pro B60Windows · 2026.11.13.051,597
3× Arc Pro B60Windows · 2026.11.13.111,724
4× Arc Pro B60Windows · 2026.11.13.051,888
Generation · tok/splain decoding
Core Ultra 5 125H12.9 with MTP + DFlash25.1
Core Ultra 9 285H16.9 with MTP + DFlash26.0
1× Arc B58012 GB · 0.49 GiB of weights in RAM over PCIe 3.07.1
1× Arc Pro B6546.7 with MTP · Linux · 2026.11.18.0821.6
1× Arc A77031.3 with MTP · ARC_BATCH=64, the default since 2026.11.18.0822.7
1× Arc Pro B602026.11.14.0922.7
1× Arc Pro B7070.6 with MTP · Linux · 2026.11.16.0235.2
2× Arc Pro B60Linux, GPU to GPU · 11-1437.8
3× Arc Pro B60Windows · run of 2026-10-3143
4× Arc Pro B60Windows · run of 2026-10-3153
the reference laptopmeasured by usersa shorter prompt, not comparable
Qwen3.8 27B GSQ-RCO IQ3_S; the laptop's prompt is 8K, the others 4K. Each bar is one user's run, with its release. The B70's prompt is a Pro B70 on native Linux (issue #27, main 7368768 with ARC_H8_NORM=1, the norm and the H8 rotation in one kernel, the default on discrete cards since 2026-11-18 (2,890 without it); with the flash attention of PR #29, the decoded weights kept resident, the H8 GEMM in 2D messages with its prefetch; 2,105 tok/s on another B70 under Windows before the weight prefetch, issue #12); its generation bar is the same Linux B70 with the GEMV rows of joshdev-de's branch (the MTP figure through serverz, 4 drafts). The B580 (issue #37, commit 7703141, PCIe 3.0) cannot hold all of the 27B's weights beside its buffers: with ARC_HOST_KEEP_GB=0.75 0.49 GiB of them stay in RAM (1.94 tok/s generation with the old 2.24 GiB there). The several-B60 bars (issue #4) are from before the weight prefetch that took one B60 from 1,322 to 1,522 tok/s, and newer runs are coming.
  • One Pro B70 (issue #27, Linux), long prompts with main 7368768: the 27B 64K at 2,131 tok/s, 131K 1,614 (attention 31 / 48 % of the time), generation after them 30.1 / 26.3 tok/s; the 35B-A3B 64K 4,337. Qwen3.6 35B-A3B: an 8K prompt at 5,647 tok/s (2026.11.17.04), 4K 5,177 and 16K 5,445 (2026.11.17.03 / .04); generation 95.3-97.6 tok/s once its kernels are cached (2026.11.16.15). Qwen3.8 Flash Next on the same card, 46 % of its 52 GB of experts in the card's memory: 4K prompt 1,414 (2026.11.17.04, the expert gate / up through the XMX GEMM), 16K 1,145 (before it); generation 38.6 tok/s over 256 tokens after a README prompt, when the expert cache holds what the text uses (6.6 in the 64-token benchmark, which starts with it cold; the rest of the experts come from RAM).
  • One Arc B580 12 GB (issue #37, Linux, PCIe 3.0, commit 7703141), 4K prompts: Qwen3.5 4B 7,915 tok/s and generation 92.6, Qwen3.5 9B 5,341 and 56.5, Qwen3.6 35B-A3B 2,469 and 84.7. The 27B by default (commit 438dfc4, 1.02 GiB of its weights in RAM): 1,783 and 3.53 tok/s.
  • One Arc A770 16 GB (issue #35, Windows): Qwen3.5 4B's 4K prompt 4,821 tok/s through the Arc 140T preset (release 2026.11.18.02, the default since 2026.11.18.03), Qwen3.5 9B 2,969-3,017 (2026.11.17.08). Two A770s, no peer access: the 27B's prompt 1,126 against one card's 1,169 and generation 12.2 against 12.6-12.8, the sums' host round trips 73.5 ms of an 81 ms token (ARC_TP_EVSUM=1, opt-in since 2026.11.18.05, is for that).
  • One B60 with MTP: the 27B generates 51.6 tok/s, the 35B 82.4 and 111.9 (release 2026.11.13.11). The 35B's 4K prompt: 3,536 tok/s (2026.11.14.09).
  • On Windows the B60s have no peer access, so several cards add their sums through host memory. On Linux two B60s (issue #17, PCIe 4.0 x8 each) sum straight from GPU to GPU: a 4K prompt at 1,630 tok/s against 1,442 through host memory, generation 37.8 against 36.1, the same logits.
  • A card on a slow link sets the pace: one in an x4 slot (2.4 GB/s, issue #19, another 27B file) makes two cards half as fast as one on the prompt (620 against 1,223 tok/s).
  • Qwen3.8 Flash Next (52 GB of experts) over 1 / 2 / 4 B60s: a 4K prompt at 665 / 1,202 / 1,491 tok/s, generation 5.7 / 34.4 / 32.7 (2026-11-13).
  • DeepSeek-V4.1-Flash (ISTA's IQ2_XS) next to 1 TB of RAM: generation 10.3 / 18.0 / 16.6 / 17.6 tok/s over 1 / 2 / 3 / 4 B60s (release 2026.11.14.09). Unsloth's UD-Q8_K_XL over 4 B60s with 32 GiB of its experts per card kept in RAM (the default since 2026.11.16.16): a 512-token prompt at 24.0 tok/s, generation 9.25 (3.55 without).
  • Panther Lake's Arc B390 (issue #1, Core Ultra X7 358H, 128 GB of DDR5-5600): the 27B's 4K prompt 258-268 tok/s on the plain XMX path, 546-560 with H8 on the wide-tile GEMM (the default since 2026.11.17.01), KL 0.0018 between the two, 583-592 with 16 GB of decoded weights resident as well (the default since 2026.11.17.02), 634-637 on release 2026.11.18.06; the 35B's 4K prompt 1,370 and generation 26 tok/s, 46 with MTP.
Details: docs/multi-gpu.md, docs/xe2-tuning.md. Have a card? bench\website_bench.ps1 and bench\decode_try.ps1 print everything needed; post the output on issue #4.
4× Arc Pro B60, long prompts64K prompt131K prompt262K promptGeneration after 131Kafter 262K
Qwen3.6 35B-A3B MoE2,15230 s1,2671.7 min7006.2 min44.537.0
Qwen3.8 27B1,25852 s9702.3 min6406.8 min32.229.0
Qwen3.8 Flash Next MoE94869 s7782.8 min5268.3 min26.922.4

One run per model and length on a user's four Arc Pro B60s (issue #4, Windows, sums through host memory), release 2026.11.14.05, 2026-11-14: runz --gpus 4 --bench-pp N --n 32. Prompt in tok/s, the middle of three passes, with the time it took; the passes spread most for the 35B at 64K (1,857-2,187, the first pass the slowest) and 131K (1,230-1,360), and for the 27B at 262K (634-662). Generation is 32 tokens right after the prompt. Files: 35B UD-IQ4_XS, 27B and Flash Next GSQ-RCO IQ3_S.

Against llama.cpp

2.7 to 7.4 times faster prompts on the same file. And a lower perplexity.

Same GGUF files, same laptop, llama.cpp's Vulkan and SYCL back ends. Every matrix product of a prompt runs on the iGPU's int8 DP4a units, with activations quantized per row and checked against an f32 reference.

Qwen3.8 27B, 2K prompt

tok/s, higher is better

inferzero113
llama.cpp SYCL25
llama.cpp Vulkan23

Qwen3.8 27B, 34K-token code prompt

seconds to the first token, lower is better

inferzero327 s
llama.cpp SYCL1,624 s
llama.cpp Vulkan2,418 s

Qwen3.5 4B, 2K prompt

tok/s, higher is better

inferzero911
llama.cpp SYCL337
llama.cpp Vulkan242
Perplexity, same file, lower is betterinferzerollama.cpp Vulkanllama.cpp SYCL
Qwen3.5 4B6.0566.1416.138
Qwen3.5 9B5.4175.4995.497
Qwen3.8 27B4.2744.3114.310

Core Ultra 5 125H, 32 GB, 2026-10-25. After the 34K prompt the 27B generates 3.7 tok/s, against 2.1 (Vulkan) and 1.1 (SYCL). How each number was measured: docs/before-after.md.

Results

Every model on one laptop, from 128 tokens to 131K. With its error against f32.

One script measures it all, bench/website_bench.ps1: prompt speed and time to the first token at each size, generation at that depth, the drafting modes, and the KL divergence of the next-token distribution from an independent f32 CPU implementation. The limit is 0.015 for every model, and about 0.02 for Flash Next's fast mode.

Model8K prompt32K prompt131K promptGenerationWith draftingKL vs f32
Qwen3.6 35B-A3B MoE8519.6 s63751 s3246.7 min21.638.1MTP + DFlash20.0040
Qwen3.8 Flash Next MoE20540 s1743.1 min14615 min6.37.6DFlash20.0117limit ~0.02
Qwen3.8 27B13561 s1254.4 min8127 min5.112.9MTP + DFlash20.0005
Gemma 4 26B-A4B MoE23235 s1244.4 min4450 min19.8–0.0048
Qwen3.5 9B48717 s43575 s2787.8 min11.821.6MTP0.0014
Gemma 4 E4B70712 s49466 s2229.8 min19.141.8drafter0.0027
Qwen3.5 4B8969.2 s70547 s3666.0 min21.035.8MTP0.0010

Core Ultra 5 125H, 32 GB, Windows 11; commit 281b083, 2026-11-09 (the 27B's 131K row and Gemma's KL measured again 2026-11-10). Prompt in tok/s with the wait for the first token; generation in tok/s for a 256-token coding answer; KL over 480 tokens of USAGE.md. Drafting never changes the text: the drafts are checked by the full model, so the answer is the one plain decoding gives.

1× Arc Pro B60, a user's run of 2026-11-108K prompt32K prompt131K promptGenerationWith draftingKL vs f32
Qwen3.6 35B-A3B MoE3,6522.2 s2,71012 s1,3021.7 min56.9109.0MTP0.0056
Qwen3.8 27B1,0228.0 s80941 s4205.2 min23.752.0MTP0.0011
Gemma 4 12B64413 s37787 s14415 min38.073.1drafter–
Qwen3.5 9B3,0952.6 s2,56913 s1,4251.5 min52.183.3MTP0.0025
Qwen3.5 4B4,6301.8 s3,19610 s1,4301.5 min64.2128.9MTP0.0020

Arc Pro B60 (24 GB), the same bench/website_bench.ps1 run by a user on issue #4; commit 97d225b, 2026-11-10. Older than the card's speed now: it predates the weight prefetch, on by default since release 2026.11.14.09, and a run on release 2026.11.14.12 has been asked for; this table will be replaced by it as a whole. The 35B is the UD-IQ4_XS file. Gemma 4 12B's KL is left out: that day's check read the wrong rows for it (fixed 2026-11-11).

1× Arc Pro B60, release 2026.11.14.094K prompt16K promptGeneration
Qwen3.8 27B1,5222.7 s1,35912 s22.7
Qwen3.6 35B-A3B MoE3,5361.2 s–73.8

The same user's card, 2026-11-14 (issue #4): each row's 4K prompt and generation from one runz --bench-pp 4096 --n 128, the 27B's 16K from --bench-pp 16384 --n 0 in the same session. A second 27B run gave 1,524 and 23.3. Generation is 128 tokens after a 4K prompt, so it does not compare with the 256-token column above. The prefetch took the 27B's 4K prompt from 1,322 tok/s (release 2026.11.14.05) to 1,522.

What it does

Built for long documents and coding agents. On hardware that was never meant for it.

An OpenAI-, Anthropic- and Ollama-compatible server with a web UI, in one 8 MB download. No compiler, no Python, no CUDA.

Mixture of experts

Models larger than the GPU's memory

Only the experts in use stay in GPU memory. The next layer's missing experts are read from the SSD while the GPU computes the current one, so Flash Next's 52 GB of experts run on a 32 GB laptop. The launcher can pack such a model so its experts are stored once, not twice: Gemma 4 26B takes 14.8 GB instead of 27.7.

Qwen3.6 35B-A3B experts cached on the laptop83 %
context with the automatic setting131,072 tokens
Speculative decoding

Faster generation, the same answer

The model's own MTP block and a DFlash2 drafter guess up to 7 tokens; the full model checks them in one pass. Accepted drafts are the tokens plain decoding would have picked.

35B, coding answer21.6 → 38.1 tok/s
27B on one B6023.2 → 49.3 tok/s
Deterministic

The same logits, bit for bit

Run after run, whatever the expert cache size. Every speed change is held to that, or to a measured KL below 0.015.

Caches

Conversations that do not start over

A new message only processes itself. After a 20K-token tool result, a one-line follow-up on Gemma 4 E4B takes 0.8 s instead of 38 s.

Documents

Long files, read properly

Whole when the context takes it, else by outline, pages or search. OCR turns scanned PDFs into Markdown, tables as tables. Answers name their page.

NPU

The laptop's NPU reads part of the prompt

With --npu on, the Core Ultra's NPU (Intel AI Boost) computes part of each long prompt chunk's matrix products while the GPU computes the rest, through Level Zero, without OpenVINO. Dense models gain; mixture-of-experts and small models gain nothing, so they leave it off by themselves. Generation does not change. Measured on a Core Ultra 5 125H; newer NPUs (Lunar Lake, Panther Lake) are untested: bench\npu_try.ps1 measures yours.

Qwen3.8 27B, 4K prompt133 → 153-162 tok/s
Gemma 4 12B, 4K prompt+10 %
27B, KL against f320.0028-0.0031
Web UI

A chat that searches, reads and listens. At 127.0.0.1:8090, or from your phone.

The server ships its own web UI: chats with files and projects, web search, a voice conversation, and tabs for transcripts and meetings. Through a Cloudflare Tunnel with a password it works from anywhere, installable as an app.

Web search and pagesThe model searches and reads pages itself; the server asks Brave, Bing, DuckDuckGo and Wikipedia at once and merges their results, public addresses only.
Voice conversationThe microphone stays open; it answers aloud sentence by sentence and can be interrupted.
Transcripts and meetingsQwen3-ASR or Whisper large-v3 write recordings down, with speakers and summaries.
ProjectsFolders with their AGENTS.md, memory and MCP servers.
For agents

Speaks the APIs your tools already use. Plugs into Pi with one command.

Prompt and session caches mean an agent's growing context is processed once. Tool calls are constrained to their JSON schema while they are sampled, so they stay well formed. Two sessions can share one model, their tokens computed in the same pass.

OpenAIPOST /v1/chat/completions · /v1/completions · /v1/models · /v1/audio/transcriptions
AnthropicPOST /v1/messages · /v1/messages/count_tokens
OllamaPOST /api/chat · /api/generate · GET /api/tags
Toolsweb_search · web_fetch · web_view · remember · forget
# any OpenAI client works the same way
curl http://127.0.0.1:8090/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "35b", "stream": true,
       "messages": [{"role": "user",
                     "content": "Summarize docs/architecture.md"}]}'
Get started

Three steps, about five minutes. Plus the model download.

The launcher ticks the model that suits your RAM, downloads it with resume and a checksum, and opens the web UI when the server is ready.

Download

From the latest release, inferzero-<version>-win64.zip for Windows or inferzero-<version>-linux-x64.tar.gz for Linux. Unpack it anywhere.

Start

Windows: double-click zero-launcher.cmd (Windows may ask once: "More info", then "Run anyway"). Linux: run ./zero-launcher.sh in a terminal.

Pick a model

Choose one and download it; the server starts when it is ready. The web UI is at 127.0.0.1:8090.

SystemWindows 11, 64-bit, with Intel's graphics driver. Or Linux (Ubuntu 24.04 and newer; users run CachyOS too) with Intel's compute runtime (nothing else: no Python): unpack inferzero-<version>-linux-x64.tar.gz and run ./zero-launcher.sh (a terminal menu, no window), or build from source with ./build.sh. Docker and Kubernetes files are in docker/ (not tested yet). The headless server only: no speech recording or desktop window there yet.
GPUCore Ultra with Arc graphics (Meteor Lake, Lunar Lake, Arrow Lake H, Panther Lake), or an Arc B-series or A-series card.
Memory16 GB for the 4B and E4B, 32 GB for the 12B to 35B models.