Loom Model Performance

Methodology

What each figure on this page means and how it was collected. This is a reference for the metrics, not a recipe for reproducing them.

Campaign resources

Approximate, from the campaign's session logs and card data (Nov 6–9, 2026). Subagents: median 2 concurrent (maximum 16), counting each subagent from its first to its last transcript event, per minute, over the minutes when the orchestrator or any subagent was active. Local GPUs: 1 (one Strix Halo; from Nov 8 only one worker used it at a time). DevLab: up to 6 concurrent jobs (the concurrency limit).

Metrics

All requests are greedy (temperature 0) chat completions with thinking off, sent identically to both engines. Each prompt starts with a unique nonce, so no prefix cache applies. Documents are real text (Loom documentation markdown) from bench_common.py (DOC_SIZES). The reported value is the mean of the runs for decode and prefill and the median of the runs for 4 streams. Each metric uses 1 to 3 runs per session (2 for decode and prefill and 3 for 4 streams in the main sessions).

decode

Generation speed after a short prompt: a request for a 300-word essay with max_tokens 300. Loom: (tokens − 1) / (end − first token) measured by the client. llama.cpp: the server's reported predicted tokens per second.

decode 2k

The same, after a ~2,000-token document, with max_tokens 256.

decode 32k

The same, after a ~32,000-token document, with max_tokens 128.

prefill 1k, 2k

Prompt tokens divided by time to first token, for ~1,000- and ~2,000-token documents with max_tokens 4. Each run takes the median of 5 samples. Loom: client-side streaming time to first token. llama.cpp: the server's reported prompt tokens per second (excludes HTTP and tokenization).

prefill 8k, 16k, 32k

The same for ~8,000-, ~16,000- and ~32,000-token documents, one sample per run. A document longer than the model's context is not measured (n/a).

4 streams

Aggregate decode tokens per second of 4 concurrent requests (400 tokens each, a history-of-lighthouses essay), counted only inside the window in which all 4 are decoding. Client-side for both engines. Loom serves 4 rows; llama.cpp runs with -np 4.

A decode request that stops before half of its max_tokens is retried once and otherwise recorded as missing. Before each decode measurement, an untimed warm-up request of the same shape runs. Scripts: perf.py, c4_steady.py, devlab_perf.sh. Speculative decoding follows one rule per model, the same on Loom and both llama.cpp builds: the model's own MTP head, else a drafter published for that exact model by its maker or Z-Lab, else no drafting (never n-gram). Each engine is loaded once for the single-stream rows; the 4-stream row loads both engines with drafting off. Each engine uses its own draft depth (Loom tuned; llama.cpp the drafter's documented --spec-draft-n-max or its default). A row within ±3% of the better llama.cpp build is a tie. Full rules: methodology.md.

Engines

Loom. loom-serve with hrx-loom 0.1.0 (hrx-system v0.1.0, commit d01d1c1, the PyPI wheel), as recorded in each card's info.json sources. Each card sets memory_bytes (the memory loom-serve may use, from 5 GB to 106 GB depending on the model) and its drafting method and depth.

llama.cpp ROCm. Prebuilt releases from lemonade-sdk/llamacpp-rocm (gfx1151, Ubuntu), fetched by fetch_llamacpp.sh. The cards record builds b1355 and b1368 to b1371; b1369 and b1370 cover 44 of the 51 cards. Server: -ngl 99 -cram 0 -kvu -np 4 (all layers on the GPU, no host-RAM prompt cache, one KV pool shared by 4 slots, pool size set per card, 33,792 to 147,456 tokens), plus the card's speculative decoding arguments where used (--spec-type draft-dflash or draft-mtp with a drafter GGUF; --spec-draft-n-max is the drafter's documented value, else omitted) and --mmproj for vision models. The exact arguments are in each card's info.json performance.baseline.args.

llama.cpp Vulkan. The official ggml-org/llama.cpp release b12211 (ubuntu-vulkan), pinned by archive sha256 9759acd2… in llamacpp-vulkan.sha256. Same workloads and speculative settings as ROCm. KV cache: per slot (4 slots of 36,864 tokens) by default, because b12211's Vulkan backend has no GPU kernel for the fused flash attention over a unified KV pool (the server falls back to the CPU), which costs 15 to 20% on decode 32k and 4 streams. DFlash servers use the unified pool because b12211 aborts a DFlash server with 4 concurrent streams on per-slot caches. One configuration per card and row (no best-of-configurations): a setting Vulkan cannot run is a “not supported” cell, never replaced by another configuration. Details: devlab_perf.sh and the differs field in each card's performance.baselines.vulkan.

Hardware

DevLab runners: AMD Ryzen AI MAX+ 395 with Radeon 8060S Graphics (gfx1151, Strix Halo), Ubuntu 24.04 (info.json sources[*].device). Every card's performance comes from DevLab devlab-perf jobs, its single source of performance numbers: for each card, Loom (served with the active wait, completion_spin_ns) and llama.cpp ROCm come from the same DevLab job on the same runner. The Vulkan numbers come from a separate DevLab job.

How the Loom models were built

Each model follows the loom-port-llm skill, orchestrated by loom-org. The source is a plain Q4_K_M GGUF from a named Hugging Face repository, pinned by sha256 in model.json.lock. The starter kit generates the Loom kernels and the command program from the GGUF's metadata, and derives the chat and tool-calling glue from the model's own chat template. The speculative decoding method and depth (or none) are chosen by measured decode speed (draft/tune.py). The package is published to the loom-fun organization on Hugging Face with the programs_sha256 of the generated programs recorded in info.json next to each measurement and gate result.

Gates

kernel_tests

Unit tests of the generated kernels against reference implementations.

golden

llama.cpp reference recorded on the same GGUF: greedy tokens and top-5 log-probs, short prompts and 1k to 32k documents.

compare

Greedy output vs the llama.cpp reference, token by token; every divergence is a near-tie.

vision

Image input vs llama.cpp with the same projector: identical prompt tokens, replies equal up to near-ties.

consistency

Concurrent requests give the same output as the same requests run alone.

spec

Speculative decoding on gives exactly the output of speculative decoding off.

agent

Multi-turn tool-calling loop: exact prompt rendering, history reuse, valid tool calls, output vs llama.cpp.

agent_think_off

The tool-calling loop with thinking off.

sampling

temperature, top_k and seed behave as specified; first-token distribution matches llama.cpp.

needle

Needle-in-a-haystack prompt at up to half the context, compared with llama.cpp; over-capacity prompt refused.

needle_full

Needle-in-a-haystack prompt at the full context is served.

decision

Decision answers (choice, score, noul probabilities) vs llama.cpp on the same prompts (decision models only).

A gate that does not apply to a model (for example vision for a text-only model) shows as –; a gate with no result shows as ?.

Features

✓ yes: the Loom package has the feature. ❌ no: the upstream model has the feature and the Loom package lacks it. – n/a: the upstream model does not have the feature. ? : not determined. The package's chat features are derived from the upstream chat template, so a feature absent from the template is absent upstream. A drafter the upstream model offers but the package does not enable is shown as available but disabled on the Hugging Face card.

Summary figures

Models: the number of cards whose info.json loaded. Architectures: the number of distinct GGUF architectures among them. Median prefill 32k speedup: for each card, Loom's prefill 32k divided by the selected baseline's on DevLab values; the figure is the median of those ratios over cards that have both. 3 days: the campaign's elapsed time, Nov 6–9, 2026.