ZharfAI Team
An extraction model returns a perfectly formed maintenance order. The JSON parses, every required field exists, the urgency is one of the allowed enum values, and the timestamp matches the requested format. The payload is still wrong: the model copied an asset identifier from an earlier attachment, interpreted “next shutdown” as today, and proposed an action the requesting technician may not authorize.
Nothing about syntactic validity proves that a value is true, current, supported by evidence, or permitted to cause an effect. Yet once an output looks like ordinary application data, teams often stop treating it as model output. A typed object crosses a trust boundary wearing a uniform.
This field guide addresses one reader decision: when may a machine consume an AI-generated payload, and when must it reject, repair, abstain, or escalate? The answer is a versioned output contract with independent structural, semantic, evidence, policy, and effect gates. Constrained generation can improve the first gate. It cannot replace the other four.
“Structured output” hides several different promises:
| Promise | What it establishes | What it does not establish |
|---|---|---|
| Valid JSON | The bytes can be parsed as JSON | The expected fields, types, or meaning |
| Schema-valid instance | The instance satisfies one validator and dialect | Business truth, freshness, or authorization |
| Constrained generation | The decoder restricts which tokens may be emitted | Semantic correctness or source support |
| Typed application object | A language binding accepted the payload | Safe use in HTML, SQL, a shell, or a tool |
| Valid tool arguments | The call matches the tool’s input shape | Whether this actor may invoke it now |
The JSON Schema 2020-12 overview identifies a specific dialect and meta-schema. Its validation vocabulary defines structural assertions such as types and enums, while noting that format may be an annotation rather than an enforced assertion. Even a standards-conforming validator therefore needs an explicit dialect, vocabulary support, and configuration.
Provider features are narrower again. OpenAI’s Structured Outputs guide documents a supported subset, schema-processing latency, and exceptional responses such as refusal or incomplete generation. Google’s Gemini guide explicitly says to validate values in application code because syntactically correct output can remain semantically wrong. Amazon Bedrock’s structured-output documentation lists another subset, rejects unsupported constructs, and documents first-use grammar compilation. “JSON Schema supported” is not a portable binary capability.
The sources support four facts: JSON Schema is dialect- and vocabulary-specific; model providers implement subsets; constrained decoding has edge cases and compilation costs; and downstream output requires security treatment. OWASP LLM05:2025 describes insufficient validation and context-aware encoding before model output reaches browsers, databases, files, or commands as improper output handling.
The proposed five-gate contract below is ZharfAI analysis derived from those facts. It is not a standard defined by JSON Schema, OpenAI, Google, AWS, or OWASP. It deliberately separates concerns that a single valid: true flag cannot represent:
The payload may advance only when the current gate passes. A later gate cannot retroactively repair a missing earlier guarantee.
An output contract should name the entire interpretation boundary:
contract_id and immutable contract_version;Store the contract version inside or beside every payload. Bind it to the AI release passport, because a changed model, prompt, decoding engine, schema, validator, or consumer can change the deployed behavior independently.
Do not silently translate one provider’s unsupported schema into a weaker shape. Compile each canonical contract into a provider profile, then test that profile against the application validator. If a numeric boundary cannot be enforced during generation, record it as an application gate; do not drop it from the contract. If portability matters, continuously run the same contract corpus through every eligible provider rather than assuming that matching API labels mean matching behavior.
Many semantic failures begin with a convenient schema. A required string encourages the model to invent a value when the source is silent. An optional field makes “not found,” “not applicable,” “redacted,” “conflicting,” and “generation failed” indistinguishable. A default can turn missing evidence into a valid business instruction.
Prefer a discriminated result:
{
"status": "supported | absent | ambiguous | conflicting",
"value": "string or null",
"evidence_refs": ["source fragment identifiers"],
"reason_code": "controlled vocabulary or null"
}
Use integers for minor currency units, explicit currency codes, normalized instants plus source time zones, controlled units, and identifiers validated against authoritative systems. Keep display text separate from machine commands. A Persian explanation and an English explanation may differ in language; the asset identity, amount, evidence, and allowed action must not.
Reject unknown properties at trust boundaries where the consumer does not understand them. Preserve the raw response separately for investigation, but never merge unfamiliar fields into an execution object. Avoid coercion such as "1,000" to 1000, a non-empty string to true, or an unzoned local time to UTC. Coercion hides the exact defect that the contract should expose.
Constrained decoding is valuable. It reduces parser failures, prevents many illegal keys or enum values, and makes retry behavior more predictable. It is still part of generation, not an independent witness.
Use this sequence:
The independent validation is not a vote of no confidence in one provider. It preserves a stable consumer contract across providers, SDKs, streaming modes, cached grammars, and future releases.
Schema validation sees local shape. Domain validation must inspect relationships and current state. A maintenance instruction can be well typed while naming a retired asset, placing inspection_completed_at after work_started_at, requesting shutdown after the maintenance window, or combining a hazard class with an incompatible procedure.
Write deterministic checks for:
Do not send arithmetic, identifier lookup, or a policy table back to another model when ordinary code can decide it exactly. The model may propose; the validator decides whether the proposal satisfies the contract.
Evidence is also typed. A reference should resolve to an immutable document version and fragment, with retrieval time, source class, and authorization. A citation-shaped string that resolves nowhere fails the evidence gate. This extends model context engineering: the output must preserve which context supports which value, not merely return a plausible answer after seeing context.
A valid command is not an authorized command. Tool name, arguments, user identity, delegated subject, tenant, purpose, approval, limits, and current state belong in a separate policy decision. The model must not set trusted fields such as approved, role, tenant_id, policy_version, or idempotency_key; the application derives them from authenticated state.
Apply the tool-permission boundary after semantic and evidence validation. Then canonicalize allowed arguments and cross the effect boundary through a narrow adapter. Parameterize database operations, encode for the destination context, allowlist file locations and network targets, and never pass generated SQL, shell, HTML, or URLs directly to an interpreter.
For consequential writes, carry the logical intent and stable action key described in the retry and idempotency guide. Validate before authorization; authorize before dispatch; verify the authoritative result after dispatch. Re-validating a payload does not prove that an external action did or did not occur.
Automatic repair is acceptable only when the defect class and allowed transformation are explicit. Removing a Markdown fence, assembling a complete buffered stream, or mapping a deprecated enum through a versioned table may be deterministic. Asking a model to “fix the JSON” can change business meaning while making the parser green.
Preserve the original bytes, validation errors, repair method, repaired bytes, and attempt identifier. Never let repair add evidence, broaden authority, choose a missing identifier, or substitute a default consequential value. Cap repair attempts under the original deadline and cost budget. If a required fact is absent or conflicting, abstain or escalate to an accountable reviewer with the source and failure reason.
Use distinct terminal dispositions: accepted, rejected_structure, rejected_semantics, rejected_evidence, rejected_policy, abstained, expired, and indeterminate_effect. This makes monitoring actionable and prevents every failure from becoming a generic regeneration loop.
Consider an illustrative bilingual maintenance service. A technician uploads an inspection note and asks the assistant to prepare—not execute—a work order. The model may emit asset_ref, finding, severity, requested_window, procedure_ref, evidence_refs, and proposed_action. It may not emit authorization or a final command.
The structural gate rejects unknown fields, truncated responses, incompatible contract versions, and illegal enums. The semantic gate resolves asset_ref in the authenticated site, confirms that the procedure applies to the equipment revision, checks the maintenance window and severity logic, and forbids shutdown for an observation-only finding. The evidence gate requires the asset photo or inspection fragment for every finding. The policy gate checks whether this technician may draft that work class and whether a supervisor must approve it.
Only then does deterministic application code build a canonical draft. A human sees the original note, the proposed fields, source fragments, validator results, and any ambiguity. Approval creates a new authorized intent; it does not mutate the model’s proposal in place. Dispatch uses a stable action key, and the maintenance system returns the authoritative work-order identifier.
If the Persian note says «تا توقف بعدی صبر شود»—wait until the next shutdown—the model cannot fill a concrete date from linguistic plausibility. It must return ambiguous unless a scheduled shutdown record is retrieved and cited. Natural language remains useful, but system state determines the command.
Classify every change:
| Change | Compatibility decision |
|---|---|
| Add an optional display-only field | Possibly backward compatible; test old consumers |
| Add an enum value | Breaking for exhaustive consumers unless negotiated |
| Make a field optional | Potentially breaking because absence gains meaning |
| Change unit, timezone, or identifier namespace | New major contract |
| Tighten a semantic rule | New validator release and replay assessment |
| Rename a field with the same description | Breaking transport change |
| Change description while preserving shape | Potential model-behavior change; evaluate |
Run old producers against new consumers and new producers against old consumers with fixed fixtures and recorded raw outputs. During a migration, accept an explicit version range and transform through reviewed adapters. Never infer a version from which fields happen to be present.
Descriptions are part of generation behavior even when they are only annotations to a validator. Treat their edits like prompt changes. Connect every deployed producer, schema, validator, adapter, policy, and consumer through an audit-ready evidence trail.
The 2025 JSONSchemaBench study evaluated constrained-decoding systems across 10,000 real-world schemas and separated efficiency, constraint coverage, and output quality. That separation is the useful lesson for application teams: a high schema-valid rate is only one measurement.
Build a contract test corpus containing:
Measure terminal completeness, canonical schema-valid rate, semantic rejection rate, unsupported-evidence rate, policy rejection rate, repair attempt and escape rate, version mismatch, validator disagreement, effect verification, and escaped downstream defects. Slice by contract version, provider release, language, tenant, task, input source, and risk tier.
Do not let an AI payload become application state or an external action until the team can answer yes:
Revisit the gate when a provider changes its supported subset, a schema or description changes, a validator upgrades, a consumer starts coercing values, a new language or source enters the workload, or a payload gains a path to external action. The goal is not to make models write prettier JSON. It is to ensure that machine-readable uncertainty never becomes machine-executed confidence.
format annotation and optional assertion behavior.
An operating guide for deciding when an AI call may be retried, hedged, failed over, or reconciled—and for preventing timeout ambiguity from becoming duplicate business action.
Read More
A practical architecture for deciding when an AI answer may be reused, which context must enter the cache key, and when freshness, authority, or consequence requires a new result.
Read More
Install and secure OpenClaw, configure its Gateway, models, workspaces, channels, memory, skills, plugins, browser tools, subagents, tasks, and automations.
Read MoreGet in touch with our team to discuss how we can help your business.