Status: Draft for discussion. Last updated 2026-11-03. All initial open questions are decided (§12). Items marked [ASSUMPTION] are defaults chosen so we can move forward and may be revised.
contio is a Rust desktop app and background service for running local LLMs on a Framework 13 (AMD Ryzen AI 300). It should make it easy to:
- Find a model from a large, well-organized catalog (family → model → format → quantization).
- Choose where it runs: CPU, iGPU, or NPU (and later, hybrid).
- Set it up automatically: download the backend engine and model weights, verify them, and configure everything in the background.
- Serve it through one OpenAI-compatible API endpoint that agentic tools can use (Claude Code-style CLIs, Aider, Continue, opencode, LangChain, …).
- Benchmark models, quants, and devices against each other, covering both speed and quality.
- Training or fine-tuning (LoRA adapters can be loaded, but not trained).
- Multi-machine or cluster serving.
- Writing our own inference kernels. contio orchestrates existing engines; it does not replace them.
- Windows or macOS support. Linux (Fedora) comes first. Keep the code portable where it costs nothing.
| Component | Detail | Relevance |
|---|---|---|
| CPU | AMD Ryzen AI 9 HX 370: 12C/24T (4× Zen 5 + 8× Zen 5c), AVX-512 | llama.cpp CPU backend; thread pinning to the Zen 5 cores matters |
| iGPU | Radeon 890M: RDNA 3.5, 16 CU, gfx1150 |
Vulkan (RADV) or ROCm/HIP |
| NPU | XDNA 2 (~50 TOPS), amdxdna driver loaded, /dev/accel/accel0 present |
Needs XRT plus an NPU-capable runtime (see §5.3) |
| Memory | 128 GB LPDDR5x, unified. Currently split 64 GB VRAM carve-out (GPU) and 64 GB system RAM (CPU/NPU) | Two memory pools to track separately (§4.3, §7.2). Bandwidth (~120 GB/s) is shared and caps decode speed |
| OS | Fedora 44, kernel 7.2, GNOME, Rust 1.99, podman available |
Key performance fact: token generation is memory-bandwidth-bound. A rough ceiling is tok/s ≈ bandwidth / bytes-read-per-token. Example: an 8B Q4 model (~5 GB) tops out around 20–25 tok/s on any device. MoE models are the sweet spot for this machine. With 64 GB per pool, large MoE models fit: e.g. a 30B-A3B model at Q8 easily, and 100B+ MoE models at ~4-bit only just (a 120B MoE at MXFP4 is ~63 GB, right at the GPU pool's limit; a split between GPU and CPU pools is an option). The GUI should show this estimate before download.
┌────────────────────┐ ┌──────────────┐ ┌──────────────────────┐
│ contio-gui │ │ contio (CLI) │ │ Agentic tools │
│ (desktop app) │ │ │ │ (OpenAI SDK clients) │
└────────┬───────────┘ └──────┬───────┘ └──────────┬───────────┘
│ Management API (REST + WebSocket events) │ OpenAI API
└──────────────┬───────────────┘ │
▼ ▼
┌───────────────────────────────────────────────┐
│ contiod (daemon, systemd --user service) │
│ ├─ Gateway / Router (axum) │
│ ├─ Model Lifecycle Manager (load/unload/TTL) │
│ ├─ Job Queue (downloads, installs, benches) │
│ ├─ Catalog + Registry (SQLite) │
│ ├─ Hardware Probe + Fit Estimator │
│ └─ Benchmark Runner │
└───────┬──────────────┬──────────────┬─────────┘
▼ ▼ ▼
llama-server llama-server NPU runtime
(CPU build) (Vulkan/ROCm) (FLM / Lemonade / OGA)
child processes on 127.0.0.1:<ephemeral port>
- The daemon owns all state. The GUI and CLI are thin clients, so closing the GUI never stops the API.
- Backends run as child processes with their own OpenAI-ish HTTP server. contio proxies to them, which keeps crashes isolated and backends swappable.
- One public port (default
127.0.0.1:8080[ASSUMPTION]). Requests are routed by themodelfield. - Everything long-running is a job: it is cancellable, resumable, reports progress, and streams events to the GUI.
crates/
contio-core/ # domain types: catalog, artifacts, devices, config, errors
contio-hw/ # hardware detection (CPU flags, GPU, NPU, memory, power sensors)
contio-catalog/ # registry, HF Hub client, fit estimation
contio-backends/ # Backend trait + llama.cpp / NPU adapters, installers
contio-jobs/ # job queue, downloads (resume, sha256), progress events
contio-api/ # OpenAI-compatible gateway + management API (axum)
contio-bench/ # perf + quality benchmark harness
contiod/ # daemon binary
contio-cli/ # `contio` CLI binary
contio-gui/ # desktop GUI binary
| Concern | Choice | Notes |
|---|---|---|
| Async runtime | tokio | |
| HTTP | axum + reqwest (rustls) | SSE streaming passthrough |
| Storage | SQLite via sqlx (or rusqlite) | catalog cache, installed artifacts, bench results |
| Config | TOML (~/.config/contio/config.toml) |
|
| Data dir | ~/.local/share/contio/{backends,models,bench} |
Models stored by content hash and deduplicated |
| Logging | tracing + tracing-subscriber, with per-backend log files | |
| GUI | Tauri 2 (Rust core + TypeScript frontend) | Decided, see §9.2 |
Family e.g. Qwen3, Llama 3.x, Gemma 3, Mistral/Devstral, Phi-4, gpt-oss, DeepSeek, GLM
└─ Model e.g. Qwen3-30B-A3B-Instruct (params, active params, ctx length, license, modality,
│ capabilities: tools, reasoning, vision, embeddings, FIM/code)
└─ Format GGUF | ONNX (OGA) | FLM | safetensors
└─ Quant Q2_K … Q8_0, IQ2_XXS … IQ4_XS, Q4_K_M, UD-*, MXFP4, F16/BF16, int4-AWQ (NPU)
└─ Artifact concrete file(s) from a source repo, size, sha256, uploader
Each Artifact declares a compatibility set: which (backend, device) pairs can run it. The GUI filters the catalog by the device you select. For example, picking NPU only shows NPU-ready artifacts.
- Curated registry (
registry/*.toml, in the repo and updatable): hand-picked families and models with good defaults, recommended quants, chat-template fixes, and tool-calling notes. This is the "well-structured" core. - Live Hugging Face Hub search: any GGUF/ONNX repo can be added. Metadata is parsed from GGUF headers (arch, params, ctx, quant type) and repo cards. These appear as "community/unverified".
- Local import: point at an existing
.gguffile or directory.
For each artifact, given the device and context length:
weights_bytes + kv_cache(ctx, layers, kv_heads, head_dim, kv_quant) + runtime_overhead- Compared against the pool the device uses: the GPU pool (VRAM carve-out, plus GTT if enabled) or the system pool (CPU/NPU). Pool sizes are detected at runtime (
mem_info_vram_total,/proc/meminfo), never hardcoded, since the BIOS split can change. - For models too large for one pool, suggest a GPU+CPU split (llama.cpp
--n-gpu-layers/ MoE expert offload) and estimate its speed. - Output: ✅ fits /
⚠️ tight / ❌ won't fit, plus an estimated tok/s from the bandwidth model, refined later with real benchmark data.
- Filters: capability (tools, vision, code, reasoning), size, license, device, "fits on my machine".
- Sort by: popularity, recency, estimated speed, benchmark score (once measured).
- Quant picker shows size, expected quality loss (heuristic tiers), and fit badge side by side.
trait Backend {
fn id(&self) -> BackendId; // e.g. "llamacpp-vulkan"
fn devices(&self) -> &[DeviceKind]; // Cpu | Gpu | Npu | Hybrid
fn supports(&self, a: &Artifact) -> bool;
async fn install(&self, ctx: &JobCtx) -> Result<InstalledBackend>;
async fn spawn(&self, a: &Artifact, opts: &LaunchOpts) -> Result<RunningInstance>; // returns local URL
fn capabilities(&self) -> BackendCaps; // tools, embeddings, vision, json-schema, ...
}- CPU: llama.cpp built with AVX-512 / Zen 5 flags. Options: thread count and core affinity (prefer the Zen 5 cores), mlock, mmap.
- GPU: llama.cpp Vulkan (RADV) as the default (verified in M0: builds in ~3 min, fastest device on every model tested). ROCm/HIP was benchmarked against Vulkan on gfx1150 (
docs/spikes/m0-gpu.md§8): 3–9% faster prefill, 5–8% slower decode. Vulkan stays the default; ROCm is an optional second llama.cpp build (source or podman), chosen per model or as the default GPU API. - Default device policy (from M0,
docs/spikes/m0-gpu.md): GPU for chat/agentic models (MoE models like Qwen3.6 35B-A3B are the sweet spot). NPU for embeddings, Whisper, and small helper models, so the GPU stays free. CPU for overflow and small models. Speed estimates use measured numbers keyed by power profile. The power profile is a speed/energy dial: in M0, balanced (~25 W cap) gave +40% prefill vs power saver (~15 W cap) at +57% J/token on the GPU. contio exposes this as a user choice ("quiet / efficient" vs "fast") instead of hiding it. - Build preflight (Vulkan):
cmake,ninja-build,gcc-c++,vulkan-headers,vulkan-loader-devel,glslc,spirv-headers-devel. Missing packages are shown as onednf installline before the build starts. - Unified memory: the iGPU uses the BIOS VRAM carve-out (currently 64 GB) plus GTT, which is borrowed from system RAM. contio detects the current limits and, if needed, shows the user how to raise them (e.g.
amdgpu/ttmkernel params). It does not silently change system config. - Install strategy (decided: user-selectable): the first-run setup wizard asks the user to choose, and Settings → Backends lets them change it later, per backend:
- Build from source: compiled in the background with Zen 5 / Vulkan / ROCm flags. Best performance, one-time ~5 min build. Suggested default in the wizard.
- Prebuilt binaries: pinned upstream releases. Fastest setup.
- Podman container: isolated and reproducible, with device passthrough (
/dev/dri,/dev/accel). - Switching methods installs the new variant side by side, then switches profiles over to it. The old one is kept until the user removes it, so the two can be benchmarked against each other.
- All three install methods implement the same
BackendInstallertrait, so each backend declares which methods it supports.
- Launched with
--jinjaand the correct chat template, which tool calling needs.
- whisper.cpp (CPU / Vulkan) as a separate backend behind the same
Backendtrait, with the same three install methods. GGML Whisper models (tiny → large-v3-turbo, quantized variants) are part of the catalog.
Decided after the M0 spike (details: docs/spikes/m0-npu.md):
- The system stack is verified working: in-tree
amdxdna, firmware 1.1.2.64, XRT 2.27,xrt-smi validatepasses (47 TOPS). - Runtime: FastFlowLM (FLM), spawned as
flm serve <tag> --port N --ctx-len N --pmode …. It has an OpenAI-compatible API with streaming, tools, JSON schema, embeddings, and audio transcriptions. Lemonade's Linux NPU path is FLM underneath, so it is not used. ONNX Runtime GenAI is not used in v1. - Install: source build via FLM's
home_install.shinto contio's prefix (no sudo). Models go inFLM_MODEL_PATH. Pinned FLM version per contio release. - Catalog: imported from FLM's
model_list.json(~70 tags, incl. Qwen3/3.5/3.6-MoE-35B, gpt-oss-20b, Gemma 4, embeddings, Whisper).*-flash(1k ctx, single-turn) tags are hidden from agentic presets. - NPU scheduling: one
flm serveprocess = one LLM + optional co-loaded embedding model (--embed 1). Several FLM processes can coexist; the NPU lock is taken per request, so parallel requests share throughput. - NPU role (from M0 measurements): good decode speed for ≤4B at ~13–15 W, but slow prefill (130–450 tok/s for 4B+, 10–35 s TTFT at 4.5k tokens). Default use: embeddings, Whisper, small helper models, battery-mode chat. Not the default for agentic coding models (GPU instead, pending the GPU spike).
- License: FLM runtime is MIT; NPU kernels are closed but free for any use and redistributable unmodified. Credit "Powered by FastFlowLM".
- Hybrid (NPU prefill + GPU decode) stays a post-v1 goal, and the
DeviceKind::Hybridvariant exists from the start so it can be added without breaking anything.
- mistral.rs (Rust-native; useful for an in-process option).
- Hybrid NPU+GPU modes (via Lemonade/OGA).
- Remote passthrough: proxy a cloud OpenAI-compatible endpoint, useful for benchmarking local against cloud.
When the user selects Model X / Quant Y / Device Z and clicks Install & Run:
- Preflight: hardware probe, check driver/permission requirements (render group,
/dev/accelaccess, memlock ulimit for the NPU), fit check, disk space check. - Backend install if missing: download or build, verify checksum, smoke test (
--versionplus a tiny model load). - Model download: parallel chunked HTTP from HF with resume, sha256 verification against the HF
lfs.oid(mismatch → delete and retry), optional HF token for gated models. One writer per artifact: a download job takes a lock on the target file, and a second request for the same artifact joins the running job (M0: two concurrent writers corrupted a.partfile). - Config generation: a launch profile (ctx size, batch, threads, KV cache quant, flash-attn, GPU layers, chat template).
- Health check: spawn the backend, run a test completion, register the model under its API name.
- Done: notification. The model is now served at
/v1.
Rules:
- Jobs persist in SQLite and survive daemon restarts (downloads resume).
- Steps that need root are never run automatically in v1. contio explains why each one is needed and shows the exact command with a "copy" button, then re-checks once the user has run it.
- Post-v1 (decided): an optional small privileged helper (
contio-helper) gated by polkit, with a fixed allowlist of actions (GTT/TTM limits, memlock limits,render/videogroup membership). It never runs arbitrary commands. - Backends are version-pinned per profile, and one-click "update backend" re-runs benchmarks for comparison.
| Endpoint | Notes |
|---|---|
GET /v1/models |
Installed models, with contio metadata in an extension field |
POST /v1/chat/completions |
Streaming (SSE), tools / tool_choice, response_format (json_schema), stream_options.include_usage, reasoning content passthrough |
POST /v1/completions |
Legacy plus FIM for code completion |
POST /v1/embeddings |
Embedding models (llama.cpp --embedding; NPU candidates) |
POST /v1/rerank |
Reranker models (llama.cpp --reranking; Jina/Cohere-style request shape) |
POST /v1/audio/transcriptions |
Speech-to-text via whisper.cpp (CPU/Vulkan); /v1/audio/translations too |
| Vision | Image inputs (image_url content parts) in chat completions via llama.cpp multimodal projector (mmproj) |
POST /v1/responses |
OpenAI Responses API translation layer (stateless; previous_response_id stored in SQLite with a TTL) |
POST /v1/messages |
Anthropic Messages API translation layer (tool_use / tool_result blocks, streaming events, /v1/messages/count_tokens) |
Every API surface can be toggled in Settings → API. Default: all three enabled on the same port. They are only reachable from localhost, so enabling them costs nothing, and most agentic tools work without the user touching any settings. Chat Completions cannot be disabled because the others are translated onto it internally.
modelresolves to an installed profile, either by alias (e.g.coder,fast,default) or full id (qwen3-30b-a3b:q4_k_m@gpu).- On-demand loading: a request for a model that isn't loaded starts it automatically. The request waits (configurable timeout) or gets 503 with
Retry-After. - Memory-aware eviction: the manager keeps a separate budget per memory pool (GPU pool, system pool) and unloads LRU or idle models in the pool that needs space (idle TTL per model). Split models count against both pools.
- Pinning: mark models as "always loaded".
- Multiple models at once (decided): e.g. a coder model on the GPU, an embedding model on the NPU, and a small model on the CPU. Loads that would exceed the budget trigger eviction first, or are refused with a clear error if pinned models block them.
- Tool-call loop guard: N identical consecutive tool calls in one response (M0: Qwen3.6 repeated the same call 7×) are collapsed to one and logged. Configurable.
- Structured-output validation:
response_formatresults are validated against the schema, and violations are reported (engines don't always enforce it). - Concurrency: per-model request queue, plus a configurable number of parallel slots (
--parallelin llama.cpp).
- Decided: binds to
127.0.0.1by default. LAN exposure is an opt-in setting that requires an API key before it can be enabled, and shows a persistent "exposed on LAN" indicator in the GUI. Optional TLS (self-signed or user-provided cert) for LAN mode. - Optional API keys, even on localhost, for per-tool usage stats.
/api/catalog, /api/installed, /api/jobs, /api/instances (load/unload), /api/bench, /api/hw, /api/events (WebSocket: job progress, instance state, live metrics).
Per (model, quant, device, backend, settings):
- Prompt processing tok/s at several prompt lengths (e.g. 512 / 4k / 16k)
- Generation tok/s
- TTFT (time to first token)
- Peak memory (RSS + GPU GTT/VRAM)
- Power & energy: SoC package power via the amdgpu hwmon
PPTsensor (readable without root; RAPL needs root), batterycurrent_now × voltage_nowon battery → joules per token (important on battery) - Concurrency: throughput with N parallel requests
- Run conditions are recorded (AC vs battery, power profile, temperature, backend version) so results stay comparable. The power profile matters a lot: in M0, power saver capped the whole SoC at ~15 W, so device comparisons were power-bound. The runner can switch the profile for the run (and restore it), and results from different profiles are never shown as directly comparable.
- Decided: all four areas, each as a small, fast subset with a fixed seed so results are reproducible:
- Coding: HumanEval / MBPP style. Generated code runs only in podman sandboxes (no network, CPU/memory/time limits).
- Tool calling: BFCL-style single, parallel, and multi-turn function calls, checked via the real API path, so chat-template bugs show up. This is the most important area for agentic use. It gates an "agent-ready" badge per (model, backend): engine labels aren't trusted (M0: an FLM model labelled tool-calling hallucinated a tool result instead).
- Reasoning / math: GSM8K / MATH subsets.
- General knowledge: MMLU-Pro subset.
- Other model types: embedding retrieval (small MTEB-style subset), reranker ranking quality, and Whisper word error rate on a small audio set.
- Custom prompt sets: user-defined YAML with expected answers or regex/JSON-schema checks.
- Side-by-side arena: the same prompt sent to 2–4 models, with blind voting stored as Elo.
- Optional LLM-as-judge using another local (or remote) model.
- Repetition:
temperature: 0is not deterministic on GPU backends (M0), so each quality item runs N times (default 3) and is reported as a pass rate. - Budgets: tool-call and structured-output tests allow ≥ 2048 tokens so thinking models can finish reasoning. A separate "thinking off" variant is tested where the model supports it.
- Stored in SQLite and exportable as CSV/JSON.
- GUI views: leaderboard table, speed-vs-quality scatter, quant-ladder chart (same model, Q2→Q8), device comparison (CPU vs GPU vs NPU).
- Bench presets: "Quick" (~1 min), "Standard", "Full".
- Dashboard: loaded models, one memory bar per pool (GPU / system, showing what each model uses), live tok/s, power draw, API endpoint and copy-able config snippets for common tools.
- Catalog: browse, filter, and search (§4.4), then a model detail page with the quant picker, device picker, fit badge, and Install & Run button.
- Installed / Profiles: manage launch profiles (ctx, threads, layers, KV quant, aliases, pinning, TTL).
- Jobs: download/build progress, logs, retry/cancel.
- Playground: simple chat to test a model, with a tool-calling inspector.
- Benchmarks: run, compare, charts, arena.
- Settings: paths, port, API keys, HF token, backend versions, hardware info.
Also a tray icon [ASSUMPTION: M5] for quick load/unload and status.
The Tauri app is a client of contiod over the management API. It does not embed the daemon, so the API keeps running without the GUI. Frontend stack (decided): TypeScript + Svelte 5 (SvelteKit in SPA mode) + Vite. Chart library [ASSUMPTION]: something lightweight such as uPlot or LayerChart.
Options considered:
| Option | Pros | Cons |
|---|---|---|
| Tauri 2 (Rust + web UI) | Fast to build rich UIs and charts, mature ecosystem | Frontend in TS; webview dependency |
| iced | Pure Rust, Elm architecture, good GNOME fit (used by COSMIC) | Fewer ready-made widgets (charts, tables) |
| egui | Pure Rust, very fast to iterate, plotting via egui_plot | Immediate-mode look, less "native" |
| Slint | Declarative, polished | Licensing (GPL/royalty-free tiers) |
contio hw # hardware report
contio search qwen3 --device gpu --fits
contio install qwen3-30b-a3b --quant q4_k_m --device gpu
contio run coder # load profile/alias
contio ps | stop <model>
contio bench qwen3-30b-a3b:q4_k_m@gpu qwen3-30b-a3b:q4_k_m@cpu --preset quick
contio serve # run daemon in foreground
| # | Milestone | Scope |
|---|---|---|
| M0 | Spikes (1–2 wk) | NPU runtime viability on Fedora 44; Vulkan vs ROCm on gfx1150; GUI framework prototype |
| M1 | Headless core | Workspace skeleton, hw probe (both memory pools), llama.cpp CPU+Vulkan install (all three install methods), GGUF download, daemon + OpenAI gateway (chat/completions/embeddings/rerank/models, streaming, tools, vision), Responses + Messages translation, per-pool eviction, CLI |
| M2 | Catalog & GUI | Curated registry + HF search, fit estimator, GUI: catalog, install, dashboard, jobs, setup wizard (install method), whisper.cpp backend |
| M3 | Benchmarks | Perf harness + power, quality subsets, tool-calling suite, comparison views |
| M4 | NPU | Integrate the winning NPU runtime from M0, NPU catalog subset |
| M5 | Polish | Tray, polkit helper, NPU+GPU hybrid, RPM package + systemd user unit, Responses/Messages edge cases |
| # | Topic | Decision |
|---|---|---|
| 1 | GUI | Tauri 2 + Svelte 5 / TypeScript, a separate client of contiod |
| 2 | Backend install | User-selectable in the setup wizard (source / prebuilt / podman), can be changed per backend later |
| 3 | API surfaces | Chat Completions + Responses + Anthropic Messages, each toggleable, all on by default |
| 4 | Concurrency | Multiple models at once, memory-aware eviction per pool |
| 5 | Memory | 128 GB unified. Currently 64 GB GPU carve-out and 64 GB system. Pools are detected at runtime |
| 6 | NPU | FastFlowLM runtime; catalog imported from FLM's model list; single-slot scheduling; hybrid NPU+GPU post-v1 |
| 7 | Quality benchmarks | Coding, tool calling, reasoning/math, general knowledge. Code runs in podman sandboxes |
| 8 | Network | Localhost by default, LAN opt-in that requires an API key |
| 9 | Root actions | Copy-able commands in v1, polkit helper with a fixed allowlist post-v1 |
| 10 | Model types in v1 | Chat LLMs, embeddings, rerankers, vision (VLM), speech (Whisper) |
| 11 | Packaging | Dev builds now, RPM + systemd user unit later |
M0: done. Spikes in docs/spikes/ (NPU, GPU/CPU, GUI). ROCm vs Vulkan and the performance profile were measured later (m0-gpu §8): Vulkan stays.
M1: done (merged to main):
- Cargo workspace (§3):
contio-core,contio-hw,contio-backends,contiod,apps/gui -
Backendtrait with llama.cpp (GPU/CPU) and FastFlowLM (NPU) adapters. FLM env is built explicitly. -
ModelManager: on-demand load, per-pool budgets with LRU eviction, pinned models, idle TTL, request leases (no eviction mid-stream), crash detection, ordered shutdown. Tested against real child processes (contio-fake-backend). - Gateway:
/v1/chat/completions,/v1/completions,/v1/embeddingswith SSE passthrough,/v1/models, OpenAI-shaped errors. Management:/api/models,/api/models/{id}/load|unload. - Config file registry (
docs/config.example.toml) until the catalog exists. - Verified end-to-end on the Framework 13: GPU (Qwen3.6 35B-A3B), CPU (Llama 1B), and NPU (Qwen3 4B + co-loaded embeddings) served at the same time. Tool calls and streaming work through the gateway.
-
contioCLI:status,hw,models/ps,load,unload,chat(streaming, stdin, speed stats),--json,--url/CONTIO_URL - GUI: model list with load/unload, live state and errors; API card with OpenAI/Anthropic endpoints; shared styles in
app.css - Responses API (incl. streaming, function calls,
previous_response_idvia in-memory store) + Anthropic Messages (incl. streaming, tool use/results, thinking blocks,count_tokensestimate). Verified with the official OpenAI and Anthropic Python SDKs (scripts/e2e/sdk_test.py). - Gateway guards: identical consecutive tool calls collapsed (non-stream + stream; streamed tool calls are buffered until the end),
response_formatvalidated (non-stream), reported under acontioresponse field - Model aliases (
[[aliases]]), includingupstream_modelto address FLM's co-loaded embedding model under its own name - Background jobs (
contio-install): progress, logs, cancellation, one writer per target (duplicate requests join the running job) - Model downloads: Hugging Face GGUF with resume (HTTP Range) + sha256 (
x-linked-etag), gated repos viaHF_TOKEN; FastFlowLM models viaflm pull - Backend installers with preflight (
dnf installhint): llama.cpp source (Vulkan, ~2 min), prebuilt (GitHub release, sha256 digest, smoke test), podman (nearest availableserver-vulkan-bNimage); FastFlowLM source. Installs are discovered at startup and registered live after install. - Verified end-to-end in an isolated data dir: all four install paths, download cancel/resume, GPU/CPU chat on each llama.cpp runtime (incl. inside the container), NPU chat + embeddings on a contio-installed FLM.
M2: done (branch m2-catalog):
- Curated catalog (
registry/catalog.toml): 47 models in 13 families (chat, vision, code, embeddings, rerankers). Every GGUF quant of each repo is listed live from Hugging Face (cached 24 h on disk, stale cache used offline); draft/MTP heads, imatrix files and incomplete shard sets are filtered out. - Fit estimator: weights + projector + KV cache (exact per-token geometry from the GGUF headers in
registry/geometry.json: sliding-window, hybrid linear attention, MLA, shared-KV layers) + compute buffers against each pool's budget → fits / tight / too big. Bits per weight and a quality tier per quant. - NPU catalog from FastFlowLM's
model_list.json(XDNA2 entries), linked to curated models where they match; FLM-only models in their own family. - Hugging Face search for any GGUF repo ("community", conservative KV estimate).
- One-step install (
POST /api/catalog/install,contio install, GUI): installs a missing backend with the method from settings (llama.cpp default: prebuilt), downloads all shards + vision projector with resume and sha256, registers the model, optionally starts it. Re-installing the same files is idempotent. - Models added through the API persist (
models.json), can be changed (PATCH: context, pinning, device, TTL) and removed (optionally deleting files inside contio's model dir).config.tomlmodels are fixed and win on id clashes. - Model kinds: embedding (
--embedding) and reranker (--reranking,/v1/rerank); vision via--mmproj(also mounted for Podman). FLM embedding models run standalone on the NPU; FLM serializes several NPU processes itself. - Jobs persist (
jobs.json, last 100); jobs cut off by a restart resume automatically (downloads continue from.part); failed jobs can be retried. - GUI: navigation; Catalog (filters, family groups, fit badges, Hugging Face search, detail dialog with quant/device picker and install); Jobs (progress, logs, cancel/retry); Backends (setup wizard: method choice saved as the default, prerequisite check with copyable
dnfcommand, install); model removal on the dashboard. Deep link/catalog?model=<id>. - CLI:
search [--hf],show,install,rm,set,retry,backends [install]. - Verified on the Framework 13 in an isolated data dir: catalog/vision/embedding/reranker/NPU installs, chat + vision answer, kill -9 during a download → resumed after restart, fresh machine without backends → llama.cpp installed automatically, SDK end-to-end test (11/11).
- Speech to text: whisper.cpp backend (
whisper-serverv1.9.5, Vulkan or CPU, source install, ~2 min; installed automatically like llama.cpp)./v1/audio/transcriptionsand/v1/audio/translations(multipart, up to 200 MB; json/text/srt/vtt/verbose_json). Catalog family with 8 Whisper models, quants listed live fromggerganov/whisper.cpp.contio transcribe. Verified: WAV/MP3, German, translation (small), subtitles, OpenAI SDK. - NPU Whisper via FastFlowLM: FLM only serves Whisper co-loaded next to an LLM (
flm serve <llm> --asr 1;whisper-v3:turboalone is rejected as "non-llm"), so it'sasr = trueunder[backends.flm]plus an alias withupstream_model = "whisper-v3:turbo". Verified: 3 s per clip after load; FLM doesn't translate. - Model editor on the dashboard (context, device, pinning, idle TTL, extra args; config models read-only) and aliases through the API (
/api/aliases,contio alias, editor), saved inaliases.jsonand dropped with their model. - Fit badges for any context:
?context=on the catalog detail endpoints,contio show -c, live in the install dialog (re-suggests the quant unless one was picked). - Playground: streaming chat with reasoning, tool definitions and a tool-call inspector (send results back), image input, speed stats and the raw request; speech mode with file upload or microphone (recorded to 16 kHz WAV in the browser).
- Daemon stops its backends on SIGTERM/SIGHUP (was Ctrl-C only); per-backend pid files let the next start stop backends orphaned by a crash or
kill -9.
M3: done (branch m3-bench):
-
contio-benchperformance harness through the daemon's own OpenAI endpoint (the path clients use): prefill and decode tok/s per prompt length (prompts calibrated to the model's tokenizer with two measurements, unique nonce per prompt against prompt caches, llama.cpptimingspreferred over client timing), TTFT, N parallel requests; SoC power from thePPTsensor at 10 Hz → watts and J/token; idle power; load time; VRAM/system pool growth and backend RSS (mapped weights don't show as "used" on CPU); estimator prediction stored next to it. - Presets quick (512 tokens, 1 run) / standard (512 + 4k, 3 runs, 4 parallel) / full (+16k, 5 runs). Run conditions recorded: power profile, AC/battery, backend + path, other loaded models.
- Runs as a background job (one at a time): unloads other idle models (optional), reloads the model for load time and memory, holds a lease so it isn't evicted.
- Results in SQLite (
bench.db: runs + headline metrics), list/detail/delete API, CSV/JSON export. CLIcontio bench run|results|show|export|rm. - Tool-calling suite (
registry/bench/tool_calls.json, 23 cases written for contio): simple, choose among 11 tools, parallel, multi-turn (results, chained lookups, error results), no-call when no tool fits, JSON schema output. Type-strict argument checks; pass rate per category; agent-ready = ≥ 90% overall and ≥ 75% in every category. Requests use the model's own sampling: greedy decoding sent Qwen thinking models into 4096-token loops. - Quality subsets (all MIT, fetched with a fixed seed by
scripts/bench/fetch_subsets.py): reasoning = 50 GSM8K problems, knowledge = 56 MMLU-Pro questions (4 per subject), coding = 40 HumanEval tasks. Quick runs the first 20 items with a 4096-token budget; standard all items, full all items 3×, 8192 tokens. Truncated answers count as failures and are reported. - Coding sandbox: generated code runs only in Podman (
python:3.12-alpine):--network=none, read-only root,--cap-drop=all, no-new-privileges, 64 pids, 256 MB, 1 CPU, user nobody, 10 s timeout. Verified: network blocked, root read-only, infinite loop killed. - Power profile per run over D-Bus (
net.hadess.PowerProfiles, tuned-ppd on Fedora): performance/power-saver are held (released automatically if contiod dies), balanced is set and restored; restored after cancellation too. - GUI Benchmarks page: run form (models, suites, preset, power profile), bar charts per metric (latest run per model), speed-vs-tool-calling scatter, pass rate per quality suite, all runs with detail (per prompt length / per case or item with first failure), CSV/JSON export.
- Measured results feed the catalog: installed quants show their latest speed, tool-calling pass rate and agent-ready badge (install dialog,
contio show). - Robustness: cancelling a benchmark drops its in-flight request (the backend stops generating); shutdown cancels benchmark jobs and gives open requests 10 s; orphan cleanup skips backends whose parent contiod is still alive.
- Retrieval suite for embedding and reranker models (
registry/bench/retrieval.json, written for contio): 24 paraphrased queries over 48 documents, half of them same-topic distractors; recall@1 and MRR. nomic-embed-text-v1.5 and Qwen3 Reranker 0.6B: 92% / MRR 0.96 each (an easier first version without distractors gave 100% for both). - Speech suite: word error rate on 20 short LibriSpeech test-clean utterances (CC BY 4.0, 2.1 MB, compiled in) plus real-time factor; also runs FLM's NPU Whisper through its alias.
- "Thinking off" variant (
thinking: false→chat_template_kwargs.enable_thinking = false) for tool-calling and quality suites; recorded with the run. - Custom prompt sets as TOML (not YAML: the same format as
config.toml;docs/bench-set.example.toml): per itemcontains,not_contains,regex,json_schema(sent asresponse_formatand validated),max_tokens; optional system prompt.contio bench run <model> --set file.toml. - Arena (GUI
/arena): the same prompt to two random models from a chosen pool, answers streamed side by side, names revealed after the vote (A / B / tie / both bad); Elo (K = 32, start 1000) recomputed from all stored votes;/api/arena,/api/arena/votes. - LLM-as-judge in the arena: a third local model votes, asked twice with the answers swapped; only a consistent preference counts (else tie, against position bias). Judge votes are stored with the judge's name and rated separately from human votes. The judge answers with thinking off and a 1024-token budget (a 4B thinking judge reasoned past 4096 tokens without a verdict); ~5 s per verdict.
- Quant-ladder and device-comparison views
Findings so far:
- llama.cpp rejects requests that combine
toolswith aresponse_formatschema ("failed to parse grammar"); FastFlowLM accepts them. The gateway could drop the schema grammar or validate afterwards (M5). - Reasoning (quick, 4096 tokens): Qwen3.5 4B 65%; every failure was a cut-off, every finished answer was right. The model reaches the answer and then keeps second-guessing it (not a loop). Qwen's recommended sampling (temp 0.6, top-k 20, presence penalty 1.5) gave the same score, so the budget, not sampling, decides quick scores; standard uses 8192 tokens.
- Coding (quick, 4096 tokens): Qwen3.5 4B 55%; all 9 failures were cut-offs, all 11 finished programs passed in the sandbox. Same pattern as reasoning: quick scores measure "correct within budget".
- Speech: Whisper large-v3-turbo Q8_0 4.2% WER on GPU (real-time factor 0.26) and CPU (RTF 3.6, slower than real time).
- Tool calling (quick): Qwen3.6 35B-A3B 100% and Qwen3.5 4B 100% (both agent-ready); Qwen3 4B on the NPU 91% (fails the chained lookup: invents the user id instead of looking it up).
First measurements (Framework 13, performance profile, AC, standard preset):
| Model | Device | Decode tok/s | Prefill 512 / 4k tok/s | TTFT 512 | J/token | Load | Memory (est.) |
|---|---|---|---|---|---|---|---|
| Qwen3.6 35B-A3B Q4_K_M | GPU | 23.0 | 497 / 509 | 1.12 s | 1.55 | 5.1 s | 21.2 GiB (23.2) |
| Qwen3.5 4B UD-Q4_K_XL | GPU | 22.5 | 655 / 641 | 0.84 s | 1.82 | 1.2 s | 4.1 GiB (5.1) |
| Qwen3.5 4B UD-Q4_K_XL | CPU | 18.5 | 194 / 163 | 2.70 s | 2.56 | 1.5 s | 6.9 GiB RSS (4.1) |
| Qwen3 4B (FLM) | NPU | 19.5 | 336 / 552 | 1.53 s | 1.83 | 3.1 s | 8.3 GiB RSS |
Finding: the CPU estimate is too low (llama.cpp's CPU RSS is 1.7× the estimate); GPU estimates are within 10–25% on the safe side.
After M3 (v0.3.x):
- Editable
config.tomlmodels: changes saved as overrides (overrides.json), reset per model. - AppImage releases (GitHub Actions on
v*tags; daemon + CLI bundled); daemon control and a systemd user service from the app. - Token usage per model: every gateway response is metered as it passes (OpenAI
usage, streams included;include_usageis requested and stripped again for clients that didn't ask). Session (since daemon start, memory) and per-day totals (bench.dbusage_daily, local days). Audio requests count seconds (WAV/FLAC headers).GET /api/usage[?days=N],contio usage [--days N], dashboard column. - Statistics tab: live generation throughput (streamed deltas counted as they pass, corrected by the final
usage; 5-minute per-second window,GET /api/usage/live), requests in flight, per-model rates; daily usage as stacked columns per model over 7/30/90 days with totals per model and per day (GET /api/usage/daily?days=N). Series colours from the validated dataviz palette (light and dark). - Statistics tab, power and memory: GPU and CPU-core power (amdgpu
gpu_metricsv3.0, APUs; other layouts show memory only) and VRAM/RAM in use over the same 5-minute window (systeminGET /api/usage/live), drawn as two panels on one time axis with one crosshair; an all-time token counter across all models at the bottom of the page (refreshed every 5 s). - Performance roadmap (
docs/PERFORMANCE.md) and an Expert → Performance section per model that explains each setting's upsides and downsides. First item: speculative decoding (n-gram, MTP); 35B-A3B edits 23 → ~110 tok/s, new code +44%. - Performance roadmap, all items measured (docs/PERFORMANCE.md): draft-model speculation (dense models; refused together with n-gram, which hangs llama.cpp b12068), KV cache precision, MoE offload (
--n-cpu-moe, exact expert sizes from the GGUF tensor table, both pools budgeted), power while serving (daemon setting), preload of pinned models, benchmark workloads (edit/code/prose with draft acceptance) and a cached follow-up turn. Measured and not shipped: larger prompt micro-batch (slower at depth, GPU reset on Vulkan at 96k), mixed backends (ROCm alone takes the gain). ROCm re-measured at agent depths: +30–59% prompt reading at 32–96k (Vulkan's flash attention falls back to the CPU on gfx1150). Catalog Qwen3.6 entries use the-MTP-GGUFrepos. - Optional ROCm build of llama.cpp next to Vulkan (m0-gpu.md §8): installed from source (
GGML_HIP, GPU arch fromamdgpu-arch, ~3 min) or podman (server-rocm-bN, ~23 GB image,/dev/kfdpassed through); no prebuilt (upstream targets ROCm 10, Fedora ships 7.1). Both builds register side by side (backends/llama.cpp/<ver>-<method>-rocm). Per-modelgpu_api(vulkan / rocm / default) and a default setting (Vulkan); changing it restarts affected models on their next request. A missing build is a clear error, not a fallback. CLI:backends install llama.cpp --gpu rocm,set --gpu; GUI: Backends card + default, model editor.
Known gaps / M3 candidates
- whisper.cpp has no prebuilt Linux binaries upstream; Podman images exist but only tagged by commit (
main-vulkan-<sha>), so only the source install is offered. large-v3-turbo doesn't translate (model limitation). - NPU models have no memory estimate yet (FLM's
footprintis shown in the catalog but not used for pool budgeting). - Shallow source builds of llama.cpp report version
b1, so the llama-server web UI asset isn't fetched (the API is unaffected). - llama.cpp Vulkan on gfx1150 runs flash attention over K/V on the CPU ("missing support" warning). Benchmarked:
--flash-attn onis still +9% prefill at 4k tokens, so it stays the default. - Podman containers use the daemon's
XDG_DATA_HOMEstorage (a custom value isolates them from the user's default storage).