Cortex Search in production: five lessons for reliable RAG

Cortex Search provides retrieval primitives, but a reliable RAG product also depends on document structure, evaluation, calibrated refusal, governance, and review loops. These are five principles to apply when scoping a build.

1. Chunking is the product, and the cross-references are the chunking

A first-pass chunker may be section-aware — keeping clauses with sub-clauses and respecting heading hierarchy — and still miss the relationships that matter.

The miss was cross-references. Insurance wordings are full of "subject to the exclusions in Section IV" and "as further limited by Endorsement CG 21 47." A section-aware chunker treats those as inline text and leaves the retriever to guess that the reader needs Section IV pulled in too. The retriever does not guess well.

Parsing cross-references at chunk time and storing them as a lightweight graph lets retrieval pull related sections into the rerank set. Build this structural layer early. The same lesson from document extraction applies here: structural work compounds; clever prompting does not.

2. Building the test set is the project

A representative set of real questions, expected answers, and supporting citations is often the most valuable artifact in a RAG project.

The harness scored every prompt change, every chunker change, every reranker tweak on three dimensions:

• Answer correctness graded against the senior's answer (LLM-as-judge with a strict rubric, spot-checked weekly by a human).
• Citation precision — does the cited passage actually support the claim — judged by a second senior adjuster on a blind sample.
• Refusal calibration — does the copilot say "not in corpus, escalate" when the answer isn't there, instead of confabulating something plausible.

Two practices make this work. First, the operating team should own the harness and add questions when users flag bad answers. Second, the harness should gate production changes: no prompt or index update ships without beating the incumbent across the agreed dimensions.

If we'd treated eval as a "phase 4 deliverable" the way the original plan had it, we would have shipped a copilot that demoed well and was quietly mistrusted by month two.

3. Citation precision and refusal calibration trade off — pick deliberately

The temptation on a RAG system is to optimize for "always have an answer." Coverage research is exactly the wrong domain for that instinct. A confidently wrong coverage opinion is worse than no opinion — it generates a coverage letter that the carrier then has to walk back, and it teaches the adjusters not to trust the next twenty answers.

The dial we tuned was the rerank score floor below which the copilot refuses. Set it too low and citation precision sags because the model is reaching for marginal hits. Set it too high and the refusal rate climbs and the adjusters route around the tool.

The right threshold should produce deliberate refusals when the evidence is weak. That behavior builds trust by giving users a clean escalation path instead of a plausible hallucination.

The right floor is domain-specific. In a low-stakes internal-knowledge copilot, a 7% refusal rate is annoying. In a coverage-research copilot, it's the trust mechanism. Set it with the people who will get blamed for a wrong answer in the room.

4. Governance carry-over is the real reason to build inside Snowflake

Governance is often the deciding factor between a packaged product and a warehouse-native RAG application. When the corpus is already inside Snowflake, existing row-access and masking policies can apply to the search index without a second access model or separate audit trail.

If your sensitive corpus is already governed inside the warehouse, Cortex Search is structurally the cheapest place to put a RAG application — not because the search engine is cheaper, but because the governance work is already done. If your corpus lives in a document management system that was never properly indexed, factor in the perimeter work; that's usually where the build cost actually sits.

5. The human-in-the-loop has to feed the eval set, or it's just a complaint button

A useful “this looks wrong” workflow captures the answer, cited chunks, and a short correction, then routes the package into a review queue. Corrections should become labeled rows in the eval set so the system is tested against the patterns it gets wrong in real use.

The same pattern that worked on the Document AI side — corrections wired back into the next iteration on a cadence — worked here. Different artifact (eval rows instead of few-shot examples), same operating discipline. If the human review loop doesn't end at the eval set, you are paying senior adjusters to do data entry.

The bigger point

Cortex Search is a retrieval primitive, not a copilot. The copilot is the chunker plus the cross-reference graph plus the reranker rubric plus the eval harness plus the refusal floor plus the human-in-the-loop wired back into evals plus the row-access policies that were already there. The teams that ship a trustworthy RAG application treat all of those as first-class. The teams that treat Cortex Search as "Cortex Search plus a prompt" ship a demo and then spend two quarters explaining why the adjusters don't use it.

If you're scoping a RAG build on a sensitive corpus, the RAG implementation deep-dive walks through the canonical pattern, and the Snowflake Cortex page explains how Cortex Search fits with Document AI, Cortex Analyst, and in-warehouse LLM functions. The readiness assessment provides a fast check of corpus and governance readiness.