Aperio Home All Suites
Coding Assistant Evaluation

Engineering Discipline

Ten self-contained programming tasks that probe how well a local model performs as a coding assistant inside Aperio — from pure helper functions to cross-cutting store changes. Each task has a copy-paste prompt, a verify command, and clear pass criteria.

Originally two separate eval suites — a 3-task warm-up for gemma3:e4b and a 7-task progression for gemma4:e4b — merged into a single progressive test. Start at Task 1 and work up; the tasks reveal how deep the model's understanding goes.

Aperio Architecture (Quick Reference)

Most tasks follow these project conventions. Knowing the layers helps you evaluate whether the model understood the right place to make a change.

LayerLocationPattern
Routeslib/routes/api-*.jsmount*Routes(router, { store }) — thin Express handlers
Handlerslib/handlers/*/Pure async functions: handler(ctx, args) — independently testable
MCP Toolsmcp/tools/*.jsZod schemas → bound handlers
Helperslib/helpers/*.jsSingle-purpose utility functions
Storedb/*.jsDB abstraction (SQLite / Postgres)
Teststests/Node built-in runner: node:test + node:assert/strict

Setup

You need two terminals — one for the Aperio server on your local llama.cpp provider, and one for the chat client where you'll paste each task prompt.

Terminal A — Aperio server 
npm run start:local        # AI_PROVIDER=llamacpp, port 31337
Terminal B — chat client 
npm run chat:local

Paste each task's prompt into Terminal B. Use a fresh conversation per task when possible. After each task, run the Verify command in a third terminal (or stop the chat and use Terminal B).

Reset between runs

If you want to re-run tasks against a different model, run this reset first:

git checkout -- lib/helpers/redactSecrets.js lib/helpers/imageTokens.js lib/terminal.js \
                tests/lib/helpers/redactSecrets.test.js \
                lib/routes/api.js db/sqlite.js db/postgres.js db/index.js
rm -f lib/helpers/truncateMiddle.js lib/helpers/slugify.js lib/helpers/truncateText.js \
      tests/lib/helpers/truncateMiddle.test.js tests/helpers/slugify.test.js \
      tests/helpers/truncateText.test.js \
      lib/routes/api-health.js lib/routes/api-stats.js lib/routes/api-tags.js

The Tasks

Ordered by what they test: pure functions first, then surgical edits, Express routing, store/DB abstraction, application flow, and cross-cutting changes. Each task lists what files it touches, a difficulty badge, the prompt to paste, what to look for when evaluating, and the verify command.

Task 1 — Pure Function

slugify() Helper ★☆☆☆☆

Probes: basic code generation, string transformation, edge-case awareness

Files:

1 new file  (lib/helpers/slugify.js) + 1 test  (tests/helpers/slugify.test.js)

Paste thisIn the Aperio project, add a new helper lib/helpers/slugify.js that exports a slugify(str) function. It should convert any string to a URL-safe slug: lowercase, replace spaces and special chars with hyphens, collapse consecutive hyphens, trim leading/trailing hyphens. Also write a test at tests/helpers/slugify.test.js. Follow existing project patterns — use ES modules, the Node built-in test runner, and node:assert/strict.
What to evaluate
  • Does it handle edge cases (empty string, already-slugified, unicode)?
  • Does the test cover those edge cases?
  • Does it follow the project's ES module style (export function …)?
Verify:
node --test tests/helpers/slugify.test.js
Result:
Task 2 — Pure Function + Off‑by‑One Trap

truncateMiddle() Helper ★☆☆☆☆

Probes: length-guarantee discipline, off-by-one awareness, export style

Files:

1 new file  (lib/helpers/truncateMiddle.js) + 1 test  (tests/lib/helpers/truncateMiddle.test.js)

Paste thisCreate a new file lib/helpers/truncateMiddle.js that exports a function truncateMiddle(str, max). If str.length <= max, return str unchanged. Otherwise keep characters from the start and the end and put a single … in the middle, so that the returned string's length is exactly max. Use ES module export. Then create tests/lib/helpers/truncateMiddle.test.js using node:test and assert/strict with cases for: a short string (unchanged), an exact-length string (unchanged), and a long string (length equals max, contains …).
What to evaluate
  • Long-case output length is exactly max (the classic off-by-one trap)
  • … is one character
  • <= edge handled correctly
  • Test imports the new module and runs green
Verify:
node --test tests/lib/helpers/truncateMiddle.test.js
Result:
Task 3 — Pure Function + Word‑Boundary Logic

truncateText() Helper ★★☆☆☆

Probes: word-boundary logic, length guarantees, defensive defaults

Files:

1 new file  (lib/helpers/truncateText.js) + 1 test  (tests/helpers/truncateText.test.js)

Paste thisIn the Aperio project, add a helper lib/helpers/truncateText.js that exports truncateText(text, maxLength = 200, suffix = '…'). It should truncate at a word boundary when possible, never cut mid-word unless a word is longer than maxLength, and always append the suffix when truncation occurs. The returned length (including suffix) must never exceed maxLength. Write a test at tests/helpers/truncateText.test.js. Use ES modules and the Node built-in test runner.
What to evaluate
  • Word-boundary logic — does it avoid cutting mid-word?
  • Edge cases: text shorter than maxLength, text exactly at maxLength, text that is one long word
  • Does the test cover those edge cases?
Verify:
node --test tests/helpers/truncateText.test.js
Result:
Task 4 — Extend Existing Pattern

Add Stripe Key Rule to redactSecrets.js ★★☆☆☆

Probes: reading existing patterns, surgical edits, regex anchoring

Files:

1 edit  (lib/helpers/redactSecrets.js) + 1 test append  (tests/lib/helpers/redactSecrets.test.js)

Paste thisIn lib/helpers/redactSecrets.js, add a new rule to the RULES array that redacts Stripe secret keys. They look like sk_live_ followed by 24 or more alphanumeric characters. Replace each match with [REDACTED:stripe-key]. Then add a matching test in tests/lib/helpers/redactSecrets.test.js that follows the existing test style.
What to evaluate
  • New rule in RULES with kind stripe-key
  • Regex uses \b anchoring
  • New test added following existing style
  • Whole test file still green, no unrelated rules touched
Verify:
node --test tests/lib/helpers/redactSecrets.test.js
Result:
Task 5 — Surgical Switch Edit

Add 'mistral' Case to imageTokens.js ★★☆☆☆

Probes: surgical precision, touching only one case, commenting

Files:

1 edit  (lib/helpers/imageTokens.js)

Paste thisIn lib/helpers/imageTokens.js, add a new case 'mistral': to the switch in imageTokenEstimate that returns 1024, with a one-line comment explaining the figure. Do not change any existing case or the default behavior.
What to evaluate
  • mistral → 1024
  • All other cases unchanged
  • Exactly one case + one comment added
  • No unrelated rewrites or "improvements"
Verify:
node -e "import('./lib/helpers/imageTokens.js').then(m => { const assert = require('node:assert'); assert.strictEqual(m.imageTokenEstimate('mistral'), 1024); assert.strictEqual(m.imageTokenEstimate('llamacpp'), 256); assert.strictEqual(m.imageTokenEstimate('anthropic'), 1070); console.log('OK — mistral added, existing cases intact'); })"
Result:
Task 6 — Express Routing

Health Check Endpoint ★★★☆☆

Probes: Express routing patterns, store integration, error handling

Files:

1 new route file  (lib/routes/api-health.js) + 1 edit  (lib/routes/api.js)

Paste thisIn the Aperio project, add a GET /api/health endpoint. It should query the store to verify the database is reachable (e.g., run SELECT 1 or call a store method) and return:
{"status": "ok", "db": "connected", "uptime": <process.uptime seconds>}
If the DB is unreachable, return 503 with {"status": "error"}. Mount it via a new mountHealthRoutes(router, { store }) function in lib/routes/api-health.js, and wire it into lib/routes/api.js. Follow existing route patterns exactly (see lib/routes/api-wiki.js for an example).
What to evaluate
  • Does it catch errors properly (try/catch, 500/503)?
  • Does the route file follow the mount*Routes(router, { store }) convention?
  • Does it integrate cleanly into api.js?
Verify:
Start the server and curl the endpoint:
npm start &
curl http://localhost:3000/api/health
Result:
Task 7 — Store/DB Abstraction

Memory Stats / Dashboard Endpoint ★★★★☆

Probes: store abstraction, aggregate queries, SQLite vs Postgres split

Files:

1 new route file  (lib/routes/api-stats.js) + edits to  db/sqlite.js, db/postgres.js, db/index.js, lib/routes/api.js

Paste thisIn the Aperio project, add a GET /api/stats endpoint that returns aggregate counts from the database. It should return:
{"memories": {"total": 42, "by_type": {"fact": 20, "preference": 12, "event": 10}}, "wiki_articles": {"total": 8, "by_status": {"fresh": 5, "stale": 2, "draft": 1}}, "agent_jobs": {"total": 3}}
Add a getStats() method to the store interface in db/index.js (which delegates to the active backend). For SQLite, implement it in db/sqlite.js with real queries against the memories, wiki_articles, and agent_jobs tables. For Postgres, add a stub in db/postgres.js that throws "not implemented". Create the route in lib/routes/api-stats.js (following the mount*Routes pattern) and wire it into lib/routes/api.js. Follow existing patterns.
What to evaluate
  • Does it use the store abstraction correctly (not raw DB in the route)?
  • Does it handle the SQLite vs Postgres split?
  • Are the queries efficient (one query per aggregate vs N queries)?
Verify:
npm start &
curl http://localhost:3000/api/stats | jq
Result:
Task 8 — SQL Query + Input Safety

Tag Suggestions Endpoint ★★★★☆

Probes: prefix search, parameterized queries, input safety

Files:

1 new route file  (lib/routes/api-tags.js) + edits to  db/sqlite.js, db/postgres.js, db/index.js, lib/routes/api.js

Paste thisIn the Aperio project, add a GET /api/tags/suggest?q=<prefix> endpoint. It should query all distinct tags from the memories table that start with the given prefix (case-insensitive), ordered by frequency (most-used first), limited to 10 results. Return:
{"tags": ["deployment", "debugging", "design"]}
Add a suggestTags(prefix, limit) method to the store (SQLite implementation, Postgres stub). Create the route in lib/routes/api-tags.js and mount it in lib/routes/api.js. Follow existing patterns. If q is missing or empty, return all tags ordered by frequency (top 20).
What to evaluate
  • Does it handle empty/null prefix gracefully?
  • Is the query case-insensitive?
  • Does it use parameterized queries (no SQL injection)?
Verify:
npm start &
curl "http://localhost:3000/api/tags/suggest?q=de"
Result:
Task 9 — Application Flow

Add a --no-memory CLI Flag ★★★☆☆

Probes: understanding existing app flow, surgical CLI changes

Files:

1 edit  (lib/terminal.js)

Paste thisIn the Aperio project, the terminal chat (npm run chat:local) starts via lib/terminal.js. Add a --no-memory CLI flag. When passed, the agent session should skip preloading memories from the store. Parse it from process.argv (e.g., --no-memory as a boolean flag — present means skip, absent means load normally). Make the change minimal — don't refactor unrelated code. Follow existing patterns in the file.
What to evaluate
  • Does it find where session initialization / memory preloading happens?
  • Is the change minimal and surgical?
  • Does it handle the flag correctly (present = true, absent = false)?
Verify:
npm run chat:local -- --no-memory
Should start chat without loading memories.
Result:
Task 10 — Cross‑Cutting Change

Add Pagination Metadata to Memory List ★★★★☆

Probes: backwards compatibility, cross-cutting changes, MCP handler safety

Files:

Edits to  lib/routes/api-memories.js, db/sqlite.js, db/postgres.js

Paste thisThe Aperio GET /api/memories endpoint currently returns {"memories": [...]}. Enhance it to include pagination metadata:
{"memories": [...], "pagination": {"total": 150, "limit": 20, "offset": 0, "has_more": true}}
The store's list/recall method should return a {rows, total} object instead of just an array. Update the SQLite implementation in db/sqlite.js to run a COUNT(*) OVER() window query or a separate count. Update the Postgres stub similarly. Update the route in lib/routes/api-memories.js to include the pagination wrapper.
Critical: Do NOT break the existing MCP recall handler — that should still receive just the rows array (call .rows on the result). Follow existing patterns.
What to evaluate
  • Does it avoid breaking the MCP handler (which expects just rows)?
  • Are the count queries efficient (not a second full scan)?
  • Is the API response backwards-compatible (additive only)?
Verify:
npm test
Also check: npm run chat:local → recall some topic
MCP recall tool should still work and return formatted text, not an object.
Result:

Per‑Task Scoring Rubric

Apply these five signals to each task after running the verify command. They measure not just "does it work" but also "was the model surgical" and "did it actually verify its own work".

Signal0 = fail1 = pass
Edited the correct filewrong file / new file when edit expectedright file
Verify command passesred / errorsgreen
Stayed surgical (no unrelated rewrites)rewrote / "improved" unrelated codeminimal diff
Produced the test when askedcode onlycode + test
Ran/read the verify itself when promptedguessed / claimed without checkingactually ran / read output

Scorecard

Results fill in automatically as you click the buttons above.

TestResultNotes
1. slugify() Helper—
2. truncateMiddle() Helper—
3. truncateText() Helper—
4. redactSecrets — Stripe Key—
5. imageTokens — Mistral Case—
6. Health Check Endpoint—
7. Memory Stats Endpoint—
8. Tag Suggestions Endpoint—
9. --no-memory CLI Flag—
10. Pagination Metadata—
TOTAL: ____ / 10 passed
Scoring notes:
  • Pass — the model implemented the task correctly and the verify command passes.
  • Fail — the verify failed, the model touched the wrong file, or made unrelated changes.
  • N/A — the task doesn't apply (e.g., the prerequisite file doesn't exist). N/A tasks excluded from the total.
  • Run each task in a fresh chat when possible.
  • For the rubric signals (surgical, test, verify), judge them on top of the basic pass/fail — a task that passes the verify but rewrote half the file is a red flag.

What the Results Mean

← Your result
9–10/10
Production-ready assistant. The model understands Aperio's architecture, follows conventions, and makes surgical changes. Trust it for real development work.
← Your result
7–8/10
Strong coding assistant. Handles most tasks well. May struggle with cross-cutting changes or abstract patterns. Good for focused tasks with clear scope.
← Your result
4–6/10
Moderate ability. Good for simple helpers and surgical edits. Struggles with routing, DB changes, or multi-file work. Double-check its output.
← Your result
1–3/10
Limited coding ability. May handle one or two pure-function tasks. Not reliable for anything involving routing, DB access, or existing code.
← Your result
0/10
Not a coding assistant. This model should not be used for development work within Aperio.

What the progression reveals

The task order traces a curve from shallow to deep understanding:

pure functions (tasks 1–3) → surgical existing-code edits (4–5) → Express routing (6) → store/DB abstraction (7–8) → application flow (9) → cross-cutting changes (10)

A model that drops off after Task 5 knows JavaScript but not architecture. One that drops off after Task 8 knows the stack but struggles with cross-cutting safety. A model that passes all 10 understands the project deeply.

Next → Capability ExamThe full system benchmark — 65+ drills across every tool category. All suites →Back to all evaluation suites.