Aperio Home All Suites
Code Graph Evaluation

Code Navigation & Understanding

Ten tests that probe whether the model correctly uses code navigation tools — code_repos, code_search, code_outline, code_context, code_callers, and code_callees — to find symbols, understand relationships, and assess change impact without reading whole files. Unlike the Coding Assistant suite (which tests writing code), this tests reading/understanding an existing codebase.

Observation-based tests. Paste the prompt, watch which tools the model calls and whether it correctly interprets the results. The verify step is the model's own response — did it find the right symbol, identify the correct callers, and give an accurate answer about the code?

Setup

The Aperio codebase must be indexed. Start the server with code graph enabled, then paste prompts into a chat session.

APERIO_CODEGRAPH=on npm run start:local
# Index the repo:
node lib/codegraph/indexer.js /path/to/aperio
# In another terminal:
npm run chat:local

These tests assume the Aperio repo itself is indexed. Run the indexer before starting. Tasks reference real symbols in the Aperio codebase (lib/agent/index.js, server.js, etc.).

The Tests

Task 1 — List Repos

What's Indexed ★☆☆☆☆

Probes: calls code_repos, correctly reads results, answers from tool data not training memory

Paste thisWhat code repositories does the code graph have indexed right now? Use the code graph tools, not your training data.
What to evaluate
  • Does the model call code_repos (not guess from training data)?
  • Does it correctly list the repo name(s) and root path(s)?
  • If the index is empty, does it say so honestly?
Result:
Task 2 — Find a Function

Symbol Search ★★☆☆☆

Probes: calls code_search with a relevant query, picks the right result from the list

Paste thisFind the function in the Aperio codebase that handles the MCP "recall" tool. Search the code graph for it — don't read whole files.
What to evaluate
  • Does the model call code_search with a relevant query (e.g., "recall")?
  • Does it pick the correct result from the list (not just the first one)?
  • Does the answer reference the correct file path (likely in mcp/tools/)?
Result:
Task 3 — Read a Symbol

Source Slice ★★☆☆☆

Probes: calls code_context with a qualified name, displays the source, adds padding

Paste thisShow me the source code of the function that starts the Aperio HTTP server. Use the code graph to find it and read just that function — no full file reads.
What to evaluate
  • Does the model call code_search then code_context with the qualified name?
  • Does the source slice include the function signature and body?
  • Does it add context padding (at least 2 lines above/below) for readability?
  • Does it avoid reading the entire file?
Result:
Task 4 — File Outline

Get the Structure of a File ★★☆☆☆

Probes: calls code_outline, correctly lists symbols and their kinds

Paste thisWhat symbols are exported from the MCP index file? Outline the server.js file using the code graph.
What to evaluate
  • Does the model call code_outline on the correct file?
  • Does the outline list symbols with their kinds (function, class, const, export)?
  • Are functions listed with their line ranges?
  • Does the answer give a useful summary of the file structure?
Result:
Task 5 — Find Callers

Who Calls This Function? ★★★☆☆

Probes: calls code_callers, correctly identifies calling functions

Paste thisI'm thinking of changing the signature of the function that starts the Aperio server. Before I do, find out what calls it using the code graph — I need to know who's affected.
What to evaluate
  • Does the model call code_callers with the correct qualified name?
  • Does it list caller functions with file paths and line numbers?
  • Does it give a clear answer about the blast radius?
  • If depth is limited, does it mention the limitation?
Result:
Task 6 — Find Callees

What Does This Call? ★★★☆☆

Probes: calls code_callees, traces internal dependencies

Paste thisShow me what internal functions the MCP tool handler calls. Use the code graph to trace its dependencies — I want to understand what it delegates to.
What to evaluate
  • Does the model search → context → call code_callees on the right symbol?
  • Does it list each callee with file path and line number?
  • Does it distinguish internal vs external calls?
  • Does it summarize what the function depends on?
Result:
Task 7 — Impact Analysis

Trace a Request from Entry to Handler ★★★★☆

Probes: search → context → callers → callees chain, multi-hop reasoning

Paste thisI need to understand what happens when a user sends a "remember" request in the chat. Trace the full execution path using the code graph — find the handler, show what it calls, and show what calls it. Walk me through the chain from entry point to data store.
What to evaluate
  • Does the model chain multiple tools: search → context → callers → callees?
  • Does it correctly trace the path (e.g., chat endpoint → handler → store)?
  • Does each hop use the correct qualified name with repo prefix?
  • Does it summarize the full execution chain clearly?
Result:
Task 8 — Qualified Names

Correct Repo Resolution ★★★☆☆

Probes: uses qualified names verbatim, passes repo to follow-up tools, avoids cross-repo confusion

Paste thisFind the function called "runAgent" in the indexed repos. Show me its source and tell me which repo and file it's in. Use the code graph — don't guess.
What to evaluate
  • Does the model use the qualified name from code_search verbatim?
  • Does it pass repo to follow-up tools (code_context)?
  • Does it correctly identify the repo and file path?
  • Does it avoid guessing the repo from the path alone?
Result:
Task 9 — Transitive Callers

Multi-Hop Dependency Check ★★★★☆

Probes: depth parameter, transitive caller resolution, summary without explosion

Paste thisFind a frequently-called utility function in the codebase (like a logging helper or error formatter), then trace its callers up to 2 levels deep. I want to know who ultimately depends on it. Use the code graph with depth control.
What to evaluate
  • Does the model use a depth parameter (≥2) on code_callers?
  • Does it summarize the transitive call graph without dumping raw output?
  • Does it handle the potentially large result set gracefully?
  • Does it correctly identify the utility function first?
Result:
Task 10 — Edge Cases

Stale Index & Missing Symbols ★★★★★

Probes: handles stale index gracefully, falls back to file reads when needed, reports honestly

Paste thisSearch the code graph for a symbol that probably doesn't exist — like "nonexistentFunction123". If it's not found, try to find it by searching for "imaginaryUtility" as well. Tell me honestly what you find. Then, search for a function you know exists (like the main server start function) to confirm the index is working.
What to evaluate
  • Does it search for the nonexistent symbol and report honestly "not found"?
  • Does it avoid hallucinating a result?
  • Does it gracefully handle the missing symbol without erroring out?
  • Does it then succeed on a known-good search to confirm the index works?
  • If the index is stale, does it recognize and report the situation?
Result:

Scoring Rubric

Signal0 = fail1 = pass
Called code graph toolsguessed from training data, read full files, used grepcode_repos / search / context / callers / callees
Used qualified names correctlyreconstructed by hand, didn't pass repo, wrong formatverbatim qualified name, repo parameter passed
Returned accurate informationhallucinated symbol locations, wrong file pathscorrect file, correct symbol, correct callers
Chained tools appropriatelydid one search and stopped; missed follow-upsearch → context → callers typical flow completed
Handled missing/stale gracefullyhallucinated results for missing symbols"not found" honestly, suggested reindex

Scorecard

Click Pass/Fail as you observe each test.

TestResultNotes
1. List Repos—
2. Find a Function—
3. Source Slice—
4. File Outline—
5. Find Callers—
6. Find Callees—
7. Impact Analysis Chain—
8. Qualified Names—
9. Transitive Callers—
10. Stale Index Handling—
TOTAL: ____ / 10 passed
Scoring notes:
  • Pass — correct tool used, accurate result, qualified names/repo passed correctly.
  • Fail — guessed from training data, read full files instead of code graph, hallucinated symbols or paths, wrong qualified name format.
  • N/A — code graph not indexed for this repo, or task requires symbols not in the index.
  • Tasks 1–4 test basic tool usage. Tasks 5–6 test call graph tools. Tasks 7–9 test multi-step chaining. Task 10 tests edge case handling.
  • This suite tests code understanding, not code generation — that's the Coding Assistant suite.

What the Results Mean

← Your result
9–10/10
Production-ready code navigator. The model correctly uses all code graph tools, chains them for impact analysis, handles qualified names correctly, and reports missing symbols honestly. Trust it for codebase understanding and refactor scoping.
← Your result
7–8/10
Strong code navigation skills. Handles basic search and context well. May miss qualified name rules, forget to pass repo, or struggle with multi-hop transitive callers. Good for directed searches.
← Your result
4–6/10
Moderate ability. Can find individual symbols but doesn't chain tools for impact analysis. May guess file paths or hallucinate non-existent functions. Needs supervision for understanding dependencies.
← Your result
1–3/10
Limited code navigation ability. Rarely reaches for code graph tools. May read entire files or guess from training data. Not reliable for code understanding.
← Your result
0/10
Cannot navigate code. Does not use code graph tools or uses them incorrectly.
← Frontend DesignTest HTML implementation quality and accessibility. All suites →Back to all evaluation suites. Next → Capability ExamThe full system benchmark — 65+ drills across every tool category.