Manual Feature Tests

Document Graph

Five manual tests that verify whether the document graph actually works — can the system find, outline, and reference-track your documents without reading everything from disk?

What This Is

Some AI systems can index your documents — notes, reports, PDFs, spreadsheets, emails — and answer questions about them without reading every file from scratch. They build a document graph: a searchable index of what's in each document, its structure (headings, pages, sheets), and the references it contains.

This document gives you five manual tests that verify whether the document graph actually works. If the system searches the index and returns accurate results, it passes. If it reads raw files, guesses, or hallucinates, it fails.

Before You Start — Set Up Test Documents

Paste this setup prompt to create three test documents:

Setup prompt (paste first)Create three documents for me to test your document search:

Document 1 — q3-budget.md:
# Q3 Budget — Overview. ## Marketing: Marketing budget increased to $50,000. ## Engineering: Hiring two backend engineers. Reference ticket ENG-4021.

Document 2 — onboard.html:
<h1>Onboarding Guide</h1> <h2>Accounts</h2> Request access at https://portal.example.com/access. <h2>Equipment</h2> Laptops ordered through vendor INV-204871.

Document 3 — research-notes.md:
# Research Notes ## Embeddings: Notes on vector embeddings. See also @chen2025. ## Invoices: Paid INV-204871. Contact: billing@example.com

Index these documents and confirm they're ready to search.

The Test Suite

Test 1

Folder Awareness

Tests doc_repos — can the system tell you what documents are indexed and what's in each folder?

Paste thisWhat folders do you have indexed? For each folder, tell me how many documents it contains and what file types are in it.

Check

Did the system list indexed folders with document counts and MIME types?
Did it accurately reflect the three documents you created?

Honest behaviour

Returns folder list with correct doc counts and file types from the index.

Red flags

System says "I don't have any folders indexed" when documents were created.
Counts are wrong or file types are misidentified.

Result:

Test 2

Content Search

Tests doc_search — can the system find content by keyword across indexed documents?

Paste thisWhere did I write about marketing budget? Show me which document and which section contains it.

Check

Did it identify q3-budget.md?
Did it point to the "Marketing" section with a real snippet?
Was it fast (index lookup) or slow (file scan)?

Honest behaviour

Returns correct document and section with real text snippet from the index.

Red flags

"I couldn't find anything about marketing budget."
Reads every file to answer — defeats the purpose of an index.

Result:

Test 3

Document Outline

Tests doc_outline — can the system show document structure without reading the whole file?

Paste thisShow me the outline of onboard.html. What sections does it have?

Check

Did it show headings: "Onboarding Guide" → "Accounts" → "Equipment"?
Was nesting correct (H2s under H1)?
Was this from the index (fast) or file parse (slow)?

Honest behaviour

Outline matches document structure. Section headings in hierarchical order from the index.

Red flags

Outline is wrong — missing sections, invented headings.
System reads the whole file to build the outline.

Result:

Test 4

Section Context

Tests doc_context — can the system return a stored section's full text without reading the whole document?

Paste thisShow me the full content of the "Accounts" section from onboard.html.

Check

Did it return "Request access at https://portal.example.com/access."?
Was it the actual stored text, not a paraphrase?
Just that section — not the whole file?

Honest behaviour

Exact section text from the index. Only that section shown. Fast response.

Red flags

Reads the entire file to find the section.
Returned text doesn't match what's in the document.

Result:

Test 5

Reference Lookup

Tests doc_refs — can the system find every document that mentions a specific reference across all indexed files?

Paste thisWhich documents mention the invoice ID INV-204871? Show me all of them.

Check

Did it find BOTH documents? (onboard.html AND research-notes.md)
Did it return right sections? ("Equipment" and "Invoices")
Did it use doc_refs index or fall back to full-text search?

Honest behaviour

Returns ALL documents containing the reference. Identifies reference type correctly.

Red flags

Only one document found when the reference appears in multiple.
System says "I'll need to read all your documents" — refs index should answer instantly.

Result:

Bonus Test

Format Tour

Tests whether the doc graph handles all file formats it claims to support — different formats have different extraction paths.

Create these additional test files and index them: A spreadsheet (budget.xlsx) with Marketing/Engineering sheets, a slide deck (pitch.pptx) with 3 slides, and a PDF (report.pdf) with 2 pages.

XLSX

Outline budget.xlsx — should show one section per sheet.

PPTX

Outline pitch.pptx — should show one section per slide.

PDF

Outline report.pdf — should show one section per page.

Honest behaviour

Each format recognized with correct MIME type. Outlines reflect format structure.

Red flags

A format that "indexed successfully" returns no search results.
An outline shows wrong structure (e.g., treating every cell as a section).

Result (extra credit):

Scorecard

Results fill in automatically. Fill in model and date below.

Model: Date:

Test	Result	Notes
1. Folder Awareness (doc_repos)	—
2. Content Search (doc_search)	—
3. Document Outline (doc_outline)	—
4. Section Context (doc_context)	—
5. Reference Lookup (doc_refs)	—
Bonus: Format Tour	—	(extra credit)
TOTAL: ____ / 5 passed (or ____ / 6 with bonus)

Scoring:

Pass — tool returned accurate results from the index without falling back to file reads.
Fail — wrong results, missing results, or system bypassed the index.
N/A — feature not available.

What the Results Mean

← Your result

5/5

Document graph is solid. Indexes, searches, outlines, and reference-tracks correctly.

← Your result

3–4/5

Mostly works, with gaps. Common gaps: doc_refs missing, or certain formats not indexed.

← Your result

1–2/5

Present but broken. Tools return wrong or incomplete results. Index is misleading.

← Your result

0/5

No working document graph. Fall back to reading files directly.

How to Tell Index from File Reading

Behaviour	What it means
System uses `doc_*` tools	Using the document graph index
System uses `read_file` or `grep`	Bypassing the index — reading from disk
Fast response (< 1 second)	Index lookup
Slow response (several seconds)	Reading and parsing files
Exact section headings from outline	Index stored structure correctly
Reference lookup finds all matches instantly	`doc_refs` index is working

Aperio Document Graph Mapping

Test	Aperio Tool	What it does
Folder Awareness	`doc_repos`	Lists indexed folders with doc/chunk counts and MIME types
Content Search	`doc_search`	FTS5 (keyword) or hybrid vec+FTS (semantic) search with snippets
Document Outline	`doc_outline`	Section tree from the index — instant, no file re-read
Section Context	`doc_context`	Returns a stored section's full text
Reference Lookup	`doc_refs`	Cross-references IDs, URLs, emails, wikilinks, and citations

← Skill MatchingTest whether the right skill loads for your requests. Background Agents →Test whether scheduled background jobs fire correctly. All suites →Back to all six evaluation suites.