Similarity is not relevance — relevance requires reasoning

The foundational insight behind PageIndex: vector embeddings measure semantic distance, but distance and relevance are different things. A document about “interest rates” might be semantically similar to a query about “monetary policy effects on housing,” but the actually relevant section lives inside a chapter on domestic policy implications — something only structure navigated by reasoning can find.

PageIndex proves this by building a tree index (like a table of contents optimized for LLMs) and having the LLM reason through it like a domain expert would — “this question about interest rates is probably under Monetary Policy.” The Mafin 2.5 system (powered by PageIndex) achieved 98.7% accuracy on FinanceBench, significantly outperforming traditional vector-based RAG systems. This is the same principle driving Agentic search beats RAG for live codebases: the agent reasons about where to look rather than relying on similarity to surface the right chunks. It also connects to why Context is the product, not the model — the quality of retrieval depends on how you structure context, not which model you use for embeddings.

The same pattern keeps winning well beyond documents, which is what makes this a rule rather than a PageIndex result (the cross-domain synthesis draws on PageIndex, Claude Code’s agentic search and @arscontexta’s skill graphs). For code search, Claude Code abandoned RAG and vectors for grep/glob — Agentic search beats RAG for live codebases. For agent skills, Skill graphs enable progressive disclosure for complex domains replaces monolithic SKILL.md files with interconnected knowledge that agents navigate by relevance. The common thread: flat structures searched by similarity hit a ceiling. Complex domains need hierarchy (trees, graphs, directories) and reasoning (LLMs deciding where to look) rather than embedding distance (cosine similarity, nearest neighbors). This is why Files are the universal interface between humans and agents is so powerful — plain files support hierarchy through directories and linking, giving both humans and agents navigable structure. A comparative analysis of 10+ knowledge graph architectures points the same way: Knowledge systems need dual-layer storage — narrative depth and structured queries can't share a format and Navigation beats search for knowledge retrieval — let each data source keep its native query interface.