ZharfAI Team
Retrieval-augmented generation can make an answer sound grounded while hiding three different failures. The search may have missed the decisive document. The prompt may have contained the document but the model ignored or contradicted it. Or the answer may faithfully repeat a source that is outdated, unauthorized, or wrong. Calling all three “RAG quality” makes diagnosis nearly impossible.
A production system needs a chain of evidence from knowledge ownership through retrieval, context construction, generation, citation, and user outcome. Each link has a different metric and a different owner. The language model is important, but it cannot repair a source catalogue with no effective dates, an access filter applied after retrieval, or an evaluation set that never asks the questions users actually ask.
Begin with the decision or task the system supports. List authoritative source types, owners, audiences, update frequency, jurisdictions, and the conditions under which an answer should abstain. Decide whether drafts, email, tickets, comments, archived policies, and user-generated notes are evidence or merely discovery aids.
Create a source hierarchy for conflicts. A signed current policy may outrank a recent chat; a regional procedure may override a global default for that region; a product manual may not authorize a financial exception. Encode these rules in metadata and context selection instead of hoping the model infers institutional authority from prose.
Every indexed item should carry a stable identifier, version, owner, effective and expiry dates where relevant, access scope, source system, language, and ingestion time. Knowledge quality is a product operation, not a one-time embedding job. Our guide to enterprise knowledge management and search covers the organizational layer in more detail.
Apply identity, tenant, role, geography, matter, and document-level permissions before candidates enter the retriever. Post-filtering a generated answer is too late: restricted text may already be present in context, logs, caches, or model output. Preserve source access controls during ingestion and invalidate indexes when permissions change.
Use separate indexes or strong filter enforcement when trust boundaries require it. Test with canary documents visible to one role and forbidden to another. Verify vector search, keyword search, reranking, cache hits, conversation memory, and citation previews. A correct final answer does not excuse unauthorized retrieval.
Log identifiers and policy decisions needed for audit without copying confidential passages into general telemetry. Treat retrieval traces as sensitive because a list of document titles and query terms can reveal as much as a generated summary.
Parse each format with structure intact: headings, tables, footnotes, lists, page references, and relationships between a procedure and its exceptions. OCR quality, encoding, duplicated headers, broken reading order, and missing attachments can silently damage the index. Measure extraction success by source type and language.
Canonicalize duplicates and link superseded versions. Do not let five copies of an old policy outvote one current policy in retrieval. Use checksums and source identifiers to make ingestion idempotent. When a source is deleted or access is revoked, propagate removal to chunks, embeddings, caches, and derived summaries.
Track pipeline lag, failed documents, orphaned chunks, empty extractions, metadata completeness, index version, and deletion latency. These controls align with broader data-quality observability. A green API does not mean the knowledge base is current.
Fixed token windows are a baseline, not a universal design. A useful chunk should preserve the unit needed to answer and cite: a policy clause with its exceptions, a table row with headers, a troubleshooting step with prerequisites, or a contract section with definition references. Excessive overlap creates duplicates; fragments without headings lose authority and meaning.
Consider parent-child retrieval: search smaller passages for recall, then supply a larger coherent parent section to the generator. Preserve document, section, page, and character offsets so a citation can open the exact evidence. For tables, store a representation suited to retrieval and retain the original for verification.
Evaluate chunking on questions, not aesthetics. Compare whether the relevant evidence appears in the top results, whether contradictory versions are distinguishable, and whether the context fits the model budget without truncating decisive text.
Users ask incomplete, conversational, misspelled, multilingual, and context-dependent questions. A query layer may resolve references, expand abbreviations, detect language, extract filters, or produce multiple searches. Every transformation can improve recall and can also change intent.
Keep the original query, rewritten query, detected filters, and reason codes available for debugging. Test dangerous rewrites: negation, dates, product versions, legal entities, and units. When ambiguity changes the answer materially, ask a clarifying question rather than retrieving confidently against a guessed interpretation.
Hybrid retrieval often combines lexical matching for exact identifiers with dense retrieval for semantic paraphrases. Use fusion and reranking only after measuring their incremental value. More components can increase latency and make failures harder to explain.
Retrieval quality asks whether the system found the evidence. Build a question set with one or more relevant passages, required authority, and known distractors. Metrics may include recall at k, precision at k, mean reciprocal rank, normalized discounted cumulative gain, and evidence coverage. Choose the metric that reflects the task: high recall may matter for compliance research, while first-result precision may matter for a quick lookup.
Slice by language, document type, source age, query length, access role, and rare entities. Record “no answer in corpus” cases to measure abstention. If a required passage is absent from the candidate set, the generator cannot be blamed for failing to quote it.
Traditional retrieval relevance is not identical to downstream usefulness for generation. The 2024 eRAG paper reported only small correlation between conventional relevance judgments and downstream RAG effectiveness in its studied setting and proposed evaluating documents through their use by the generator. That is a research result under specific experiments, not proof that relevance metrics should be discarded. Use both retrieval diagnostics and end-to-end measures.
These terms answer different questions:
A model can be faithful to an obsolete policy and therefore factually wrong. It can be factually correct from memorized knowledge but unsupported by retrieved evidence. It can cite a relevant document that does not support a specific number. These cases need different remediation.
RAGAS is a primary research example that operationalizes several RAG evaluation dimensions, alongside ARES. They offer useful automated evaluation ideas; their reported results do not guarantee that a judge will be reliable for a new domain, language, or risk level.
Language-model judges can review more cases than a human team, but they share model biases, can be sensitive to prompts, and may accept polished unsupported answers. Build a blinded human sample with clear rubrics and domain experts. Measure agreement, false passes, false failures, and segment-specific behavior.
Use deterministic checks where possible: citation target exists, quoted number matches the passage, access scope is valid, output schema is correct, and required sections are present. Reserve model judges for semantic questions and provide them the authoritative evidence.
Version judge model, prompt, rubric, and sampling method. When any changes, overlap old and new evaluations to calibrate the trend. Do not declare improvement from a score series whose measuring instrument changed silently.
After retrieval, reranking and context construction should remove duplicates, prefer authoritative current versions, preserve necessary neighbors, and surface conflicts. Attach source identifiers and effective dates to each passage. Allocate the context budget intentionally instead of filling it with the top k chunks regardless of redundancy.
When sources conflict, do not collapse them into a smooth answer. Explain the conflict, identify the applicable rule if metadata supports it, or escalate. When evidence is insufficient, abstain and offer the closest verifiable material. A refusal with useful next steps is better than an elegant invention.
Prompt instructions should tell the generator what constitutes acceptable evidence, how to cite it, when to abstain, and which actions require approval. They cannot substitute for correct filtering and source governance.
A citation is valuable only if a reader can verify the claim. Link to the stable source version and, when possible, the exact section or page. Display title, owner, and effective date where authority matters. Avoid citations that point only to a homepage, a huge document without location, or an inaccessible internal identifier.
Test citation coverage at the claim level, not merely whether the response contains brackets. Sample answers and ask: which claims require support, does the cited passage entail them, and can the user open it under the same permissions? Track stale, broken, mismatched, and inaccessible citations separately.
Citation interfaces also collect feedback. Let users report “source does not support this,” “policy is outdated,” or “better source exists.” Route each signal to the knowledge owner or retrieval team, not a generic thumbs-down queue.
Use three evaluation layers. A stable labeled set measures regression. A rotating challenge set covers new policies, attacks, languages, and known failures. Production telemetry measures actual task outcomes: reformulation, citation opening, correction, escalation, abandonment, and verified incident.
Public shared tasks such as the 2025 TREC RAG Track can improve comparability and research practice. They remain benchmarks under defined corpora, topics, and protocols; success there is not evidence that a private production corpus is current, permission-safe, or useful to its users.
Monitor drift in query mix, no-result rate, source freshness, retriever scores, context length, answer abstention, citation use, and human correction. Link anomalies to index, model, prompt, and policy versions. The observability principles in AI-agent monitoring are especially relevant when RAG triggers tools rather than only producing text.
A low-risk employee search assistant and a clinical decision aid should not share thresholds. Define gates by task: minimum evidence recall, maximum unsupported-claim rate, zero unauthorized retrieval, citation-support accuracy, freshness coverage, latency, and appropriate abstention. Slice critical metrics; a high global average can hide failure in Persian queries or one business unit.
Require human review or deterministic confirmation for high-impact actions. RAG can supply evidence, but retrieval confidence is not authorization. Record the source version that supported a decision so it can be audited after policies change.
Use staged rollout, a known-good index, rollback for model and prompt changes, and a switch to search-only mode. If generation becomes unreliable, users may still benefit from ranked, permission-safe evidence.
Consider an assistant that answers employee expense questions. The ingestion pipeline accepts only policy sources with an owner and effective date, parses sections and tables, and marks regional applicability. Access control is copied into the index. Superseded policies remain auditable but are excluded from current answers unless the user asks about a historical date.
At query time, the system identifies region and employment type from authorized profile data, clarifies ambiguous travel dates, and runs hybrid retrieval. A reranker prioritizes applicable current clauses. The context packet contains the main rule, exceptions, and relevant table rows with version identifiers. The generator must cite every limit and abstain if regional applicability is unknown.
The test suite separately measures evidence recall, version selection, context relevance, answer faithfulness, factual correctness, completeness, and citation support. Production monitoring adds successful task completion, correction, escalation, source feedback, latency, and unauthorized-access canaries. This architecture makes a bad answer diagnosable.
Typical failures include:
This ownership map prevents every problem from becoming “try a larger model.”
Start with fifty to one hundred real questions from one bounded workflow. Assign authoritative answers and passages, include no-answer and conflict cases, and test access roles. Establish retrieval metrics before adding generation. Then evaluate context construction, faithfulness, correctness, completeness, and citations separately.
Launch with a knowledge owner, freshness service level, feedback route, and rollback. Review failures weekly by stage. Add confirmed production cases to regression data under appropriate governance. The result will not be a system that never errs; it will be a system whose errors are visible, attributable, and improvable.
RAG earns trust when it helps users inspect the evidence, not when it makes generated text sound certain. Retrieval quality and answer faithfulness are connected, but they are not the same—and neither can rescue untrustworthy knowledge.
Source status was checked on 2026-07-30. The TREC 2025 proceedings include the TREC RAG Track and should be interpreted as benchmark research under its published protocols, not production certification. The primary research papers for RAGAS and ARES describe automated evaluation approaches for distinct RAG dimensions. The eRAG paper studies the relationship between retrieval evaluation and downstream generation. Reported research results depend on their datasets and methods; automated evaluators must be validated for the intended domain and language.
How enterprises can build permission-aware AI search with governed sources, provenance, measurable retrieval quality, controlled answers, and useful feedback.
Read More
The next generation of enterprise AI should not merely produce an answer. It should show the evidence, uncertainty, authority, and action path behind it.
Read More
Synthetic data needs provenance, purpose, validation, contamination controls, and a retirement rule. Artificial does not mean anonymous or harmless.
Read MoreGet in touch with our team to discuss how we can help your business.