System Benchmark

Capability Exam

A comprehensive 65+ drill benchmark covering every Aperio tool category — memory, wiki, code graph, files, shell, web, skills, multi-tool chains, and security guardrails. Each section below opens into a dedicated page with copy-paste prompts and pass/fail tracking.

How to run the exam: Pick a section below. Each drill page shows the exact prompt to paste into Aperio's chat and the ✅ expected tool/skill result. Click Pass when the model fires the right tool and produces the correct result; click Fail otherwise. Your scores are saved automatically in your browser and appear on the scorecard below. The full exam takes ~30–60 minutes depending on model speed.

Before You Start

Prerequisites:
  • Shell: Set APERIO_ENABLE_SHELL=1 for §5 shell drills
  • Roundtable: Configure ROUNDTABLE_AGENTS with ≥2 provider:model pairs for §11
  • Wiki refresh: Set WIKI_REFRESH_PROVIDER for §2.5
  • GitHub token: Set GITHUB_TOKEN for §6.2 to avoid rate limits
  • Code graph: Index the repo first — use the Code panel or ask the agent (required for §3)
  • Embedding backfill: After fixture import (§0), wait ~10s before semantic recall drills in §1.2

§0 — Fixture Import (Setup)

Required before any drill. Import the Maya Chen persona dataset so the memory, recall, wiki, and dedup drills have real data. The dataset is 28 fictional memories tagged aperio-exam.

Setup

Import the aperio-exam fixture

Option 1. Open your favorite terminal, ensure Aperio is running and paste the snipped below

PORT=31337
curl -s https://raw.githubusercontent.com/BaiGanio/aperio/refs/heads/master/.github/capability-exam/exam.memories.json |
curl -s -X POST "http://localhost:$PORT/api/memories/import" -H "Content-Type: application/json" --data-binary @-

Option 2. Copy the block from below and paste it within Aperio terminal or web chat

Run the Aperio exam. Fetch the exam run-book, do the setup section to import the fixture, then wait for me before each drill.
✅ Expected
  • API returns {"imported":28,"errors":[],"note":"Embeddings are being generated..."}
  • recall by tag aperio-exam returns 28 memories
  • Wait ~10s after import before running semantic recall drills (§1.2)
⚠ If it fails
  • Check your port — try 3000, 1701, or 31337
  • Ensure the Aperio server is running
  • Do not proceed to §1 until recall by tag returns 28

Exam Sections

Click any section to open its drills. Scores update as you go.

§1 Memory Tools 10 drills

remember, recall (semantic / by-type / by-tag / temporal), update_memory, forget, deduplicate_memories, backfill_embeddings, remember with TTL.

Open drills →
§2 Wiki 5 drills

wiki_write, wiki_list, wiki_search + wiki_get, staleness detection, wiki refresh with regeneration.

Open drills →
§3 Code Graph 7 drills

code_repos, code_search, code_outline, code_context, code_callers, code_callees, code_status.

Open drills →
§4 File Tools 9 drills

read_file, scan_project, write_file, edit_file, append_file, generate_docx, generate_xlsx, delete_file, pinned memory check.

Open drills →
§5 Shell 6 drills

run_shell (allowed / git / blocked operator), run_node_script, run_python_script, syntax_check.

Open drills →
§6 Web 3 drills

fetch_url, fetch_github_issue, read_image / describe_image.

Open drills →
§7 Skills Matched 17 drills

Every canonical skill prompt — pptx, xlsx, docx-advanced, pdf, canvas-design, theme-factory, doc-coauthoring, prompt-optimizer, reasoning-planning, coding-standards, skill-creator, mcp-builder, webapp-testing, memory-protocol, codegraph, tool-integration, handoff.

Open drills →
§8 Multi-Tool Chains 4 drills

Integration under load — recall→generate→file, codegraph→write→syntax→run, web→memory, recall→wiki_write→provenance.

Open drills →
§9 Guardrails 4 drills

Negative drills — path blocking, write sandbox, shell sandbox, file extension check.

Open drills →

§10 — Teardown

Cleanup

Remove the exam fixture

Paste thisRecall everything tagged "aperio-exam" and forget each one — clean up the exam fixture. Also delete any scratch/ files created during the drills, and the Nimbus wiki article if I don't want to keep it.
✅ Expected
  • Agent recalls by tag aperio-exam and forgets the set
  • Scratch files and test artifacts are cleaned up

§11 — Roundtable

Not scored

Multi-agent discussion

Requires ROUNDTABLE_AGENTS with ≥2 models and ROUNDTABLE_MAX_ROUNDS set. Skip if unconfigured.

Paste thisStart a roundtable discussion: two models should debate whether Nimbus should switch from NATS to Kafka, given what we know from memory about the original decision.
✅ Expected
  • Agent spawns a roundtable; each model responds in turn
  • Both perspectives reference the NATS decision from memory
  • Final output is a synthesized discussion with citations to source memories
Result:

Scorecard

Scores auto-populate from your per-drill pass/fail selections on each section page. Fill in the model details below.

SectionDrillsPassedScoreStatus
§1 Memory Tools10—— / 10
§2 Wiki5—— / 5
§3 Code Graph7—— / 7
§4 File Tools9—— / 9
§5 Shell6—— / 6
§6 Web3—— / 3
§7 Skills Matched17—— / 17
§8 Multi-Tool Chains4—— / 4
§9 Guardrails4—— / 4
§11 Roundtable1—— / 1not scored
TOTAL: ____ / 65 drills passed across 9 scored sections

What the Results Mean

← Your result
Excellent
≥90% drills passed. The model handles nearly all Aperio capabilities. Production-ready.
← Your result
Strong
≥75% passed. The model is reliable for most tasks. A few tool categories may have gaps — check which sections scored low.
← Your result
Partial
≥50% passed. Usable for basic workflows but expect failures in complex tool chains or specialized tools.
← Your result
Weak
<50% passed. The model struggles with Aperio's tools. Consider a different model or use only the sections that passed.

Result Template

After completing the exam, copy this template and paste it as a comment on issue #129. It auto-fills from your scorecard.

← All Evaluate SuitesReturn to the full model evaluation suite. Start: §1 Memory Tools →10 drills — remember, recall, update, forget, and more.