Can a model use Aperio's tools the right way?
This is not a generic speed test. It checks whether one local llama.cpp model can complete fixed Aperio tasks inside an 8, 16, 24, or 32 GB RAM/context tier—and records what happened when it cannot.
A report card for one local AI model
Aperio can run entirely on your own machine using a local model. This page shows how well one such model handled a fixed set of tasks — nothing is uploaded, everything stays in your browser.
Did it use the right tools, safely?
Each task checks whether the model recalled the right memory, used the correct tools in order, actually changed what it claimed, stayed within safety limits — plus how fast it ran and how much memory it used.
Compare, diagnose, decide
Load a run to see which tasks passed or failed and why. Press Load demo for an example first. A red "invalid" banner means the test setup was wrong — not the model.
Where do these files come from, and how do I load them?
This page displays the latest metrics-only export committed at docs/benchmarks/pilot/qualification-data.js. Re-run the export command after a benchmark to replace that file. The full run evidence remains private; the export deliberately excludes prompts, answers, memories, transcripts, and paths.
1 · Produce a run (person with the project checked out)
From the project folder, in a terminal, validate first, then run one isolated case. The tiers are 8, 16, 24, and 32 GB:
# check the config is valid — starts nothing:
npm run model-tier:pilot -- --validate
# run one case (example: Gemma in the 16 GB tier):
npm run model-tier:pilot -- \
--model gemma4-e4b-ud-q4kxl \
--tier 16 \
--case chain-recall-wiki \
--campaign audit-YYYYMMDD-gemma-e4b-16gb-chain-recall
A completed run writes a private copy beside the full evidence and refreshes the docs copy. For an older run, generate it separately:
npm run model-tier:export -- \
--model gemma4-e4b-ud-q4kxl \
--tier 16 \
--campaign audit-YYYYMMDD-gemma-e4b-16gb-chain-recall
What is a campaign ID? It is simply the unique label and result-folder name for a run. You can choose a descriptive one after --campaign. If you omit that option for a single run, Aperio generates a UTC timestamp such as 20260715T112103Z.
The run uses a temporary workspace and SQLite database, starts isolated Aperio and llama.cpp processes, waits for readiness, performs the case, verifies tool and state evidence, then stops the processes and removes the temporary runtime folder.
2 · Find the safe dashboard file
Every run saves them under a folder named by tier, model, and campaign ID:
var/benchmarks/model-tiers/<tier>gb/<model-id>/<campaign-id>/
The command also replaces docs/benchmarks/pilot/qualification-data.js, which this page loads automatically. It contains only model identity, case statuses and timings, tool/turn status flags, and RAM/timing samples. The original run.json, cases.jsonl, metrics.csv, logs, and transcripts stay in the private run folder for debugging.
If you supplied --campaign, look for that exact folder name. Otherwise, open the newest timestamp-named folder. On a Mac, open Finder → Go → Go to Folder… and paste the full project path plus the path above.
Never upload or commit the original run files. The viewer only needs the safe export.
3 · Load them onto this page
- Open this page after exporting; the latest committed dashboard appears automatically.
- Use the file controls only when you need to inspect a different local export.
- Private source files are not dashboard inputs; missing metrics show Unavailable.
Glossary — what the labels mean
- Case / task
- One thing the model was asked to do, e.g. "recall why we chose NATS."
- Prompt sent
- The exact message the model received for this task — the user turn it had to act on.
- Pass / Fail
- Whether the model met every check for that task. Failing one check fails the task.
- Invalid
- No normal pass/fail result was persisted. Read the reason: it may be a context limit, loop timeout, or harness/readiness problem.
- Context-limit failure
- llama.cpp explicitly reported that the request exceeded the served context window. This is not a generic timeout.
- Loop timeout
- The turn deadline expired without explicit context-limit evidence; the model may be looping or failing to converge.
- Harness / readiness failure
- The app, model, fixture, embeddings, graphs, or infrastructure never became valid. This is not a fair model failure.
- Hard gate
- A must-pass task. A model can't be recommended if it fails one of these.
- Tool order
- Did the model call the right tools (recall, fetch, remember…) in the right sequence?
- Answer terms
- Specific words the correct answer must contain (e.g. "NATS", "Postgres").
- State
- Did the model actually change something it claimed to — like saving a new memory?
- Guardrail
- A safety task: the model must refuse or be blocked from doing something out of bounds.
- llama RSS
- How much memory the local model itself used while running.
- Swap delta
- Extra "overflow" memory the machine had to use. High swap means the model is too big for that machine.
A retry never erases the first attempt. The final persisted result determines the classification, while both attempts remain useful reliability evidence. One pilot run is diagnostic evidence—not enough by itself to choose an installer default.
Qwen3.5 9B
CompleteQualification cases
Structured events and state verification decide each result—not answer text alone.
Runtime evidence
Summed RSS for runner-owned llama.cpp processes across the pilot timeline.