The Feedback Firewall: Stop AI Outputs Becoming Ground Truth

An invoice-coding assistant recommends an account. A clerk accepts it. The posted voucher is exported as a successful example, then added to the next fine-tuning set. Six months later, the team reports that the new model agrees with historical decisions more often. But many of those decisions were suggestions made by the previous model. Agreement has become a measurement of inheritance, not necessarily correctness.

This pattern reaches far beyond accounting. Search ranking determines which items receive clicks; a support copilot changes which cases escalate; generated text enters later web corpora; and a reviewer may reproduce an AI answer’s framing in a “human” label. Production can alter the evidence that future models learn from.

The practical decision is: which production events may be used for monitoring, weak supervision, training, or evaluation—and what independent evidence must exist before promotion? A feedback firewall answers that question at the data boundary. It does not ban feedback or synthetic data. It prevents unknown, model-made, and model-influenced signals from silently becoming authoritative ground truth.

Recognise four different loops

Treating every loop as “data drift” hides the mechanism. Four paths deserve separate controls:

1. Recursive content loop. Model-generated text, images, summaries, or labels return to a later training corpus as if they were independent observations.
2. Selection loop. A model decides what is shown, investigated, approved, or measured, so the future dataset contains outcomes only for the selected cases.
3. Reviewer-anchoring loop. A person sees the model output before labelling and may accept, lightly edit, or rationalise it. The final record is human-approved but not independent of the model.
4. Cross-system loop. One system consumes another model’s output through a file, feature, report, or human process; the dependency is absent from the declared pipeline.

Google researchers described direct and hidden feedback loops in the 2015 paper Hidden Technical Debt in Machine Learning Systems. A direct loop lets a model influence selection of its future training data; a hidden loop can connect otherwise separate systems through changes in the world. The paper also warns about undeclared consumers of model outputs. These are system dependencies, not defects that a better prompt can repair.

Performative Prediction, published at ICML 2020, formalises a related point: when predictions support decisions, they may change the outcomes they aim to predict. Its mathematical results depend on stated assumptions; they do not provide a universal recipe for every deployed model. The durable engineering lesson is narrower: retraining on post-deployment outcomes is not ordinary supervised learning when deployment helped create those outcomes.

Separate verified evidence from design judgment

The evidence supports several boundaries. NIST’s voluntary Generative AI Profile, AI 600-1, published in July 2024, identifies model collapse and homogenisation when training over-relies on synthetic data. It recommends documenting training-data provenance, checking possible training/test cross-contamination, verifying deduplication particularly for synthetic samples, and evaluating feedback loops between provenance signals and human reviewers. The UK NCSC’s November 2023 secure-development guidance explicitly includes user feedback among AI assets and asks teams to document sources of training, fine-tuning, human, and operational feedback.

Original research also requires careful scope. The 2024 Nature paper AI models collapse when trained on recursively generated data found that indiscriminate recursive training can make low-probability parts of the original distribution disappear in its theoretical and experimental settings. It does not establish that every synthetic example is harmful. A separate 2024 study, Is Model Collapse Inevitable?, reported that accumulating synthetic generations alongside original real data avoided divergent test error in the models and datasets it tested. Its authors explicitly do not claim an optimal universal strategy.

The feedback firewall below is ZharfAI analysis built from those findings, not a published standard or proof that one architecture eliminates all feedback risk. It turns provenance and independence into enforceable admission states so a team can see which assumptions its dataset carries.

Give every event a provenance envelope

Do not infer provenance later from a table name such as human_feedback. Attach a compact envelope when the event is created, before joins and exports erase its history.

Refer to protected evidence rather than copying sensitive content into broad analytics. The discipline complements data-quality observability: freshness checks test arrival; the envelope tests meaning.

The 2021 Datasheets for Datasets paper proposes recording motivation, composition, collection, and recommended uses. Apply that at dataset and event level: “reviewer feedback” is incomplete if it omits whether the answer was visible first.

Route signals through explicit admission states

Create an allow-list policy. A convenient event is not automatically a label.

Evaluation eligibility should be stricter than training eligibility. A noisy correction may still help a robust training process, while the same item can corrupt the benchmark that decides whether the model improved. Keep the stores and permissions separate, not just two Boolean columns in one export.

If synthetic data is useful, issue it a generation manifest and purpose-specific approval rather than disguising it as observed data. The synthetic-data birth certificate explains how to record generator, source boundary, privacy testing, utility testing, and release tier. Synthetic data can cover controlled edge cases; it cannot certify its own realism.

Preserve independent anchors and unexposed samples

A model cannot learn outcomes for options it never showed. A triage system that escalates only high-scoring cases will receive detailed investigator labels for that slice and little evidence about false negatives. A recommender receives clicks on displayed items, not counterfactual clicks on hidden ones. Naively training on the resulting records rewards the current selection policy.

Preserve proportionate evidence that the deployed model did not select or pre-label: random audit samples, shadow review, delayed outcomes, a rule-based control, or expert review before revealing the suggestion. Experimentation needs ethical, legal, and operational approval and is unsuitable when withholding action could cause harm. Where exploration is unsafe, use independent records and explicit uncertainty; do not pretend a missing counterfactual was observed.

Keep original human or physical observations available as anchors. The counter-result on accumulation is especially useful here: “synthetic is always poisonous” is too crude, but replacing scarce original evidence with recursive approximations is risky. Track performance on rare classes, minority language forms, unusual workflows, and high-cost exceptions because average accuracy can rise while the tails disappear.

Measure reviewer independence, not only reviewer identity

Human approval is not automatically an independent label. Record the order of exposure:

• blind-first: the reviewer forms a judgment before seeing the model;
• model-first: the suggestion is visible from the start;
• comparison: independent judgment is captured, then compared with the model;
• adjudicated: a qualified reviewer resolves a documented disagreement using external evidence.

Use model-first review for efficient operations when appropriate, but do not relabel its acceptances as a blind gold set. On a risk-based sample, collect blind-first judgments and reveal the model only afterward. Compare acceptance, correction size, disagreement type, review time, and downstream outcomes. This estimates anchoring and automation bias without requiring every production case to be duplicated.

Capture correction reasons—wrong entity, unsupported claim, stale policy, missing source, unsafe action, ambiguity, or defective source record—because edit size does not measure semantic error.

Seal evaluation from the learning path

Evaluation contamination can make a feedback loop look like progress. Give each benchmark item a stable identity or privacy-preserving membership marker. At dataset build time, reject exact duplicates, near duplicates where the method is reliable, shared source documents, and descendants of evaluation items. Split by time, customer, incident, document family, or workflow when random rows would leak neighbouring evidence.

Do not send release-gate failures directly into training and then rerun the same gate as if it remained independent. Promote failures into a development set, create or acquire fresh held-out cases, and version the transition. The AI release passport can bind the final training manifest, evaluation manifest, policy, model, and evidence to the assembly that was actually promoted.

For media, the April 2026 C2PA 2.4 specification provides tamper-evident assertions about origin and changes, not a judgment that content is true. Use credentials as one admission signal; absence does not prove an asset is synthetic.

Worked example: an invoice exception copilot

Consider a bilingual finance workflow that proposes an account and tax treatment for invoices that failed deterministic matching. The first model suggests maintenance expense; the clerk accepts it; the voucher posts; no one raises an exception.

That sequence contains several different signals:

The admission service stores the accepted case in the operations log but excludes it from gold evaluation. A sample enters blind review. If the governing records support the account, the adjudicated case can become training-eligible. If a later reconciliation reverses it, the reason—not merely the reversal flag—determines whether the original label was wrong. This avoids teaching the next model that “whatever the previous model got approved is truth.”

The same logic works in support, security, underwriting, moderation, and maintenance. Replace the accounting record with the appropriate independent anchor: customer-confirmed resolution, incident evidence, verified outcome, policy adjudication, or physical inspection. Consequence, rights, and domain expertise determine the gate.

Build the firewall as a one-way control plane

A practical architecture separates collection from promotion:

production events
  -> provenance enricher
  -> admission policy
       -> monitor store
       -> candidate queue -> blind/adjudicated review -> training manifest
       -> quarantine
independent anchor stream ---------------------------> evaluation manifest
training + evaluation manifests -> release passport -> promotion gate

Make decisions reproducible and idempotent. Version rules, evidence references, and state-change reasons. Dataset builders should accept immutable manifests, not arbitrary queries; training jobs must not read the evaluation store. Stable IDs propagate deletion, rights expiry, and corrections into descendants.

An audit-ready evidence trail should record who promoted a signal, under which policy, against which evidence, and into which dataset and release. Log the decision metadata without retaining unnecessary personal or confidential content.

Detect feedback theatre

Common failure modes are concrete:

• “Human approved” replaces provenance. The interface showed the answer first, but the export calls the result human-labelled.
• Completion becomes correctness. A closed ticket, posted voucher, or unreported incident is treated as a successful outcome.
• Only selected cases receive labels. The model is evaluated on the slice its predecessor chose to inspect.
• A filter erases the origin. Generated and observed records land in the same table before tagging.
• Gold failures become training examples. The benchmark is memorised through repeated repair.
• Rare cases vanish behind the average. Recursive or selection effects reduce tail coverage while aggregate agreement rises.
• Provenance becomes truth. A valid signature proves association and integrity, not factual accuracy or usable rights.
• One dashboard hides gated unknowns. High provenance coverage averages away a consequential unknown source.

Operate with metrics that reveal independence

Track denominators and slices, not a single “feedback quality” score:

• provenance completeness: events with every required envelope field divided by all captured events;
• model-exposure rate: labels formed after visible model output, by task and reviewer workflow;
• independent-anchor ratio: training and evaluation items supported by evidence not created or selected by the judged model;
• admission yield: candidates promoted, rejected, quarantined, and expired, with reasons;
• selection coverage: outcome evidence for selected and independently sampled cases;
• tail retention: coverage and error on rare classes, languages, workflows, and severe exceptions across releases;
• reviewer anchoring delta: blind-first versus model-first disagreement and correction patterns;
• evaluation-isolation violations: attempted or confirmed overlap with training sources or descendants;
• downstream contradiction rate: reversals, reopenings, incidents, or adjudications that invalidate accepted labels;
• lineage resolution time: time to identify every dataset and release containing a corrected or prohibited event.

Set hard gates for missing provenance, evaluation overlap, prohibited rights, and unresolved high-consequence contradictions. Do not dilute them into weighted averages.

Start with one feedback export feeding one model. Sample its rows, reconstruct who or what created each label, record model exposure and selection, and attempt to find independent evidence. Route the rows through the admission table before the next dataset build. The first audit usually reveals that the organisation has several useful monitoring signals, fewer defensible training labels, and a much smaller independent evaluation set.

Revisit the firewall when the interface changes, a model begins selecting cases, reviewers see new explanations, synthetic generation enters the pipeline, a downstream system adopts the output, label rights change, or a benchmark failure is promoted into development data. Feedback improves a system only when its origin, exposure, selection, and evidence remain visible. Otherwise the next model may simply become a more confident copy of its predecessor.

Source Notes — reviewed August 4, 2026

• NIST AI 600-1, Generative AI Profile — final cross-sector voluntary profile, July 2024; its official publication page was updated in April 2026.
• NCSC Guidelines for Secure AI System Development: secure development — published 27 November 2023; guidance on AI assets, supply chains, technical debt, and documented human or operational feedback.
• AI models collapse when trained on recursively generated data — Nature 631, version of record published 24 July 2024; scoped theoretical and experimental evidence about indiscriminate recursive training.
• Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data — original 2024 research reporting bounded error under its accumulation settings, not a universal optimal strategy.
• Performative Prediction — Proceedings of ICML 2020; formal treatment of predictions that influence future outcomes.
• Hidden Technical Debt in Machine Learning Systems — NIPS 2015; system-level analysis of direct and hidden feedback loops and undeclared consumers.
• Datasheets for Datasets — Communications of the ACM, December 2021; dataset documentation framework.
• C2PA Content Credentials Technical Specification 2.4 — April 2026 specification for tamper-evident provenance assertions; not a truth or quality judgment.