Manual Feature Tests · 2 categories · 10 tests

Document Graph & Vision

Ten manual tests across two dimensions: document graph — can Aperio find, outline, and reference-track your documents from its index; and vision pipeline — can the VLM read structured fields from images and pass them to the main model for document search and cross-referencing. Requires a running Aperio instance.

What This Is

Aperio indexes your documents — notes, reports, PDFs, spreadsheets, emails — so it can answer questions about them without reading every file from scratch. The document graph (Section A) builds a searchable index of content, structure, and cross-references. The vision pipeline (Section B) adds VLM-powered image extraction: attach a scanned invoice, the VLM reads the fields, and the main model uses those values to search related documents.

This document gives you ten manual tests across both categories. Section A works with any indexed documents. Section B requires sample invoice images and a configured VLM model.

Before You Start — Set Up Test Documents

Paste this setup prompt to create three test documents for Section A:

Setup prompt (paste first)Create three documents for me to test your document search:

Document 1 — q3-budget.md:
# Q3 Budget — Overview. ## Marketing: Marketing budget increased to $50,000. ## Engineering: Hiring two backend engineers. Reference ticket ENG-4021.

Document 2 — onboard.html:
<h1>Onboarding Guide</h1> <h2>Accounts</h2> Request access at https://portal.example.com/access. <h2>Equipment</h2> Laptops ordered through vendor INV-204871.

Document 3 — research-notes.md:
# Research Notes ## Embeddings: Notes on vector embeddings. See also @chen2025. ## Invoices: Paid INV-204871. Contact: billing@example.com

Index these documents and confirm they're ready to search.

For Section B, you additionally need a VLM model configured (e.g. OLLAMA_VLM_MODEL=qwen3-vl:8b), a folder of trade documents indexed (e.g. trade-docs/ with commercial-invoice.txt, letter-of-credit.txt, swift-mt700.txt), and sample scanned invoice images (scanned-invoice.png, scanned-invoice-2.png).

Section A · Document Graph

Does the index work for search, outline, and cross-references?

Each test verifies a different doc_* tool. The system passes if it returns accurate results from the index without falling back to file reads.

Test 1

Folder Awareness

Tests doc_repos — can the system tell you what documents are indexed and what's in each folder?

Paste thisWhat folders do you have indexed? For each folder, tell me how many documents it contains and what file types are in it.
Check
  • Did the system list indexed folders with document counts and MIME types?
  • Did it accurately reflect the three documents you created?
Honest behaviour
  • Returns folder list with correct doc counts and file types from the index.
Red flags
  • System says "I don't have any folders indexed" when documents were created.
  • Counts are wrong or file types are misidentified.
Result:
Test 2

Content Search

Tests doc_search — can the system find content by keyword across indexed documents?

Paste thisWhere did I write about marketing budget? Show me which document and which section contains it.
Check
  • Did it identify q3-budget.md?
  • Did it point to the "Marketing" section with a real snippet?
  • Was it fast (index lookup) or slow (file scan)?
Honest behaviour
  • Returns correct document and section with real text snippet from the index.
Red flags
  • Misses the document entirely or returns a wrong document.
  • Returns a plausible-sounding but invented snippet.
Result:
Test 3

Document Outline

Tests doc_outline — can the system tell you the section structure of a document without reading it from disk?

Paste thisShow me an outline of q3-budget.md — what sections does it have and what is each about?
Check
  • Did it return the correct sections (Overview, Marketing, Engineering)?
  • Was the response instant or did it read the file?
Honest behaviour
  • Returns the correct section tree from the index — no file re-read needed.
Red flags
  • Reads the file instead of using the outline index.
  • Outline is empty or shows wrong section names.
Result:
Test 4

Section Context

Tests doc_context — can the system retrieve a specific section's full text from the index without re-reading the file?

Paste thisShow me the full text of the "Marketing" section from q3-budget.md.
Check
  • Does the returned text exactly match "Marketing budget increased to $50,000"?
  • Was it instant or did it read the file?
Honest behaviour
  • Returns the correct section text from the index. Fast response.
Red flags
  • Reads the entire file to find the section.
  • Returned text doesn't match what's in the document.
Result:
Test 5

Reference Lookup

Tests doc_refs — can the system find every document that mentions a specific reference across all indexed files?

Paste thisWhich documents mention the invoice ID INV-204871? Show me all of them.
Check
  • Did it find BOTH documents? (onboard.html AND research-notes.md)
  • Did it return right sections? ("Equipment" and "Invoices")
  • Did it use doc_refs index or fall back to full-text search?
Honest behaviour
  • Returns ALL documents containing the reference. Identifies reference type correctly.
Red flags
  • Only one document found when the reference appears in multiple.
  • System says "I'll need to read all your documents" — refs index should answer instantly.
Result:
Section B · Vision Pipeline

Can the VLM extract fields from images and feed them into document search?

These tests verify the two-model pipeline: VLM reads structured fields from scanned documents, then the main model uses those values to search indexed documents. Requires indexed trade documents and a configured VLM model.

Test 6requires Aperio

Document Search Only (No VLM)

Tests: main model calls doc_search for a beneficiary IBAN without any VLM involvement.

Paste thisFind documents in my indexed documents that mention beneficiary IBAN DE89 3704 0044 0532 0130 00. Use doc_search and return only the filenames.
Check
  • Did the main model call doc_search?
  • Is commercial-invoice.txt among the results?
  • Was no VLM used (no image attached, no describe_image call)?
Honest behaviour
  • Main model calls doc_search and returns filenames. No VLM triggered.
Red flags
  • Main model tries to describe a non-existent image.
  • doc_search not called — model guesses or hallucinates the IBAN source.
Result:
Test 7requires Aperio

Image Extraction Only (VLM Answers)

Tests: VLM reads a scanned invoice and returns structured JSON without the main model repeating the description.

Setup: Attach scanned-invoice.png to your prompt.
Paste thisRead this attached invoice image and return only this JSON:
{"invoice_number":null,"lc_reference":null,"total_amount":null,"beneficiary_iban":null}
Use numeric-only total_amount and do not guess.
Check
  • Did the VLM answer directly without the main model repeating the description?
  • Expected: inv-2026-06-5021, lc-2026-06-001, 1266250.00, DE89 3704 0044 0532 0130 00
  • Is there exactly one answer block (not VLM description + main model re-description)?
Honest behaviour
  • Single JSON answer from VLM. Main model does not re-describe the image.
Red flags
  • Main model repeats the VLM description before outputting JSON.
  • Values are wrong or the model admits "I can't see attachments."
Result:
Test 8requires Aperio

Follow-Up Awareness

Tests: The main model remembers VLM-extracted evidence across turns without re-analysing the image.

Setup: Run test 7 first in the same chat. Do NOT attach the image again.
Paste thisWhat beneficiary IBAN was shown in that invoice? Answer with the IBAN only.
Check
  • Did the main model answer from conversation context?
  • Expected: DE89 3704 0044 0532 0130 00
  • Was the image re-analysed? (Should not be — the value is in the chat history.)
Honest behaviour
  • Answers correctly from context. No VLM re-trigger.
Red flags
  • Says "I can't see any image in this conversation" despite test 7 having just run.
  • Wrong IBAN or hallucinated value.
Result:
Test 9requires Aperio

Image + Document Tools Combined

Tests: VLM reads an LC reference from an image → main model calls doc_search with it.

Setup: Start a fresh chat. Attach scanned-invoice.png.
Paste thisRead the LC reference from this invoice image. Then use doc_search to find similar documents in my indexed documents that mention the same LC reference. Return a short table with filename, document type, and matching reference.
Check
  • Did the VLM read LC-2026-06-001 from the image?
  • Did the main model call doc_search with that value?
  • Did the main model avoid repeating the VLM description?
  • Likely matches include commercial-invoice.txt, letter-of-credit.txt, swift-mt700.txt
Honest behaviour
  • VLM extracts the LC ref; main model calls doc_search; results are accurate.
Red flags
  • Main model doesn't call doc_search — guesses which documents match.
  • VLM description is repeated by the main model before the tool call.
  • doc_search returns no results because the trade docs aren't indexed.
Result:
Test 10requires Aperio

Second Invoice Extraction

Tests: VLM extracts fields from a different invoice document (tests generalisation).

Setup: Start a fresh chat. Attach scanned-invoice-2.png.
Paste thisExtract the invoice number, date, buyer name, total, LC reference, IBAN, and BIC from this attached image. Return one compact JSON object and nothing else.
Check
  • Expected values:
    invoice_number: INV-2026-05-1189
    date: 2026-05-14
    buyer_name: Aciers Spéciaux du Rhône SA
    total: 11488.00
    lc_reference: LC-2026-05-008
    iban: DE89 3704 0044 0532 0130 00
    bic: COBADEFFXXX
  • Is the output a single JSON object with no extra commentary?
Honest behaviour
  • Correct JSON with all fields. No extra text before or after.
Red flags
  • Missing fields or wrong values.
  • Main model interleaves its own commentary with the JSON.
Result:
Bonus Test

Format Tour

Tests whether the doc graph handles all file formats it claims to support — different formats have different extraction paths.

Create these additional test files and index them: A spreadsheet (budget.xlsx) with Marketing/Engineering sheets, a slide deck (pitch.pptx) with 3 slides, and a PDF (report.pdf) with 2 pages.
XLSX

Outline budget.xlsx — should show one section per sheet.

PPTX

Outline pitch.pptx — should show one section per slide.

PDF

Outline report.pdf — should show one section per page.

Honest behaviour
  • Each format recognised with correct MIME type. Outlines reflect format structure.
Red flags
  • A format that "indexed successfully" returns no search results.
  • An outline shows wrong structure (e.g., treating every cell as a section).
Result (extra credit):

Scorecard

Results fill in automatically. Fill in model and date below.

TestResultNotes
1. Folder Awareness (doc_repos)—
2. Content Search (doc_search)—
3. Document Outline (doc_outline)—
4. Section Context (doc_context)—
5. Reference Lookup (doc_refs)—
6. Vision — Document Search Only—
7. Vision — Image Extraction Only—
8. Vision — Follow-Up Awareness—
9. Vision — Image + Doc Tools Combined—
10. Vision — Second Invoice—
Bonus: Format Tour—(extra credit)
TOTAL: ____ / 10 passed   (or ____ / 11 with bonus)
Scoring:
  • Pass — tool returned accurate results from the index without falling back to file reads.
  • Fail — wrong results, missing results, or system bypassed the index.
  • N/A — feature not available.

What the Results Mean

← Your result
10/10
Document graph and vision pipeline are solid. Indexes, searches, extracts, and cross-references work correctly.
← Your result
6–9/10
Mostly works, with gaps. Common gaps: vision extraction is unreliable, or VLM → doc_search handoff fails.
← Your result
3–5/10
Present but broken. Tools return wrong or incomplete results. Index or VLM pipeline needs attention.
← Your result
0–2/10
No working pipeline. Fall back to reading files and images directly.

How to Tell Index from File Reading

BehaviourWhat it means
System uses doc_* toolsUsing the document graph index
System uses read_file or grepBypassing the index — reading from disk
Fast response (< 1 second)Index lookup
Slow response (several seconds)Reading and parsing files
Exact section headings from outlineIndex stored structure correctly
Reference lookup finds all matches instantlydoc_refs index is working

Aperio Tool Mapping

SectionTestAperio ToolWhat it does
AFolder Awarenessdoc_reposLists indexed folders with doc/chunk counts and MIME types
AContent Searchdoc_searchFTS5 (keyword) or hybrid vec+FTS (semantic) search with snippets
ADocument Outlinedoc_outlineSection tree from the index — instant, no file re-read
ASection Contextdoc_contextReturns a stored section's full text
AReference Lookupdoc_refsCross-references IDs, URLs, emails, wikilinks, and citations
BDocument Search Onlydoc_searchMain model searches indexed docs for an IBAN from memory
BImage ExtractionVLM (describe_image)VLM reads structured fields from a scanned invoice image
BFollow-UpConversation contextMain model remembers VLM evidence across turns
BCombined PipelineVLM → doc_searchVLM extracts LC ref → main model searches indexed docs
BSecond InvoiceVLM (generalisation)VLM extracts from a different invoice document
← Honesty & RobustnessTest file-claim honesty, gullibility, and memory recall. Skill Matching →Test whether the right skill loads for your requests. All suites →Back to all evaluation suites.