The Resilience Layer: AI Risk Management in Critical Infrastructure

AI in power, water, transport, telecommunications, health, and emergency services changes physical and societal risk. A recommendation can alter maintenance priorities; a forecast can affect dispatch; an agent can touch operational technology; a false alarm can exhaust operators; and a missed alarm can interrupt an essential service. Average benchmark performance is therefore the wrong release criterion.

Critical-infrastructure AI must be engineered as part of a safety, security, and resilience system. That means explicit operating boundaries, deterministic protection, independent sensing, human authority, graceful degradation, cyber defense, recovery, and evidence that the system behaves acceptably under stress—not only on normal historical data.

Separate decision support from control

Start by classifying the AI’s authority:

Observe: summarize logs, classify alarms, or detect patterns without changing the operational state.
Recommend: propose inspection, dispatch, or response options to an authorized operator.
Prepare: draft a work order or control plan that remains unexecuted.
Execute bounded action: act inside a deterministic envelope with interlocks, rate limits, and immediate override.
Autonomous safety- or service-critical control: highest consequence and rarely an appropriate starting point for a general-purpose model.

The boundary must exist in system architecture, not prose. A language model that is “told not to open the breaker” but holds the credential can still do it. Use separate identities, least privilege, allow-listed tools, independent policy enforcement, two-person approval where warranted, and hardware or control-system interlocks.

Keep conventional protection systems independent. AI may help forecast instability, but it should not replace protective relays, emergency shutdown logic, certified control, alarms, or statutory operator duties unless it passes the full engineering and regulatory process applicable to that function.

Use a lifecycle risk framework

The NIST AI Risk Management Framework organizes work into Govern, Map, Measure, and Manage. The NIST AI RMF core stresses inventory, clear roles, continuous monitoring, third-party risk, incident response, recovery, override, and decommissioning.

As of 30 July 2026, NIST is revising AI RMF 1.0 and has started, but not completed, a critical-infrastructure profile. Its April 2026 concept note identifies operational realities such as legacy and distributed assets, deterministic behavior, explainability, graceful degradation, fail-safe operation, adversarial robustness, rigorous testing/evaluation/validation/verification, and supply-chain visibility. Because this is a concept note for work in progress, do not present it as a final standard.

Apply the lifecycle:

Govern: name accountable executives, system and safety owners, operators, model-risk staff, cyber teams, vendors, and regulators. Define risk appetite and change authority.
Map: document service mission, physical process, users, affected communities, dependencies, credible hazards, threat actors, and non-AI alternatives.
Measure: test technical performance, human factors, cyber resilience, environmental conditions, rare events, and downstream impact.
Manage: avoid, mitigate, transfer, or accept risk; monitor deployment; respond to incidents; and deactivate safely.

Build the hazard model around the service

Begin with loss, not model error. Examples:

loss of life or injury;
prolonged loss or unsafe degradation of essential service;
equipment damage or environmental release;
cascading failure across sectors;
loss of operator situational awareness;
unauthorized physical action;
discriminatory service allocation;
exposure of sensitive infrastructure or customer data.

Then trace how AI could contribute. A load forecast may underpredict an extreme event because training data lacks it. A vision model may miss corrosion under unusual lighting. A maintenance model may prioritize frequent low-cost failures over rare catastrophic ones. A retrieval agent may follow a malicious instruction hidden in a vendor manual. A model update may change alarm classification without a control-system change request.

Use established safety methods—hazard and operability studies, fault trees, failure mode and effects analysis, safety cases, bow-tie analysis—and extend them to data, model, prompt, tool, and human interaction. Connect AI risks to the facility’s existing risk register rather than creating a parallel “AI ethics” document.

Scenario: an AI assistant in an electric-grid control room

A utility deploys an assistant to correlate SCADA alarms, weather, outage tickets, asset history, and approved procedures. It may summarize the situation and suggest diagnostic checks. It cannot issue switching orders, change protection settings, or write to SCADA.

During a storm, several telemetry feeds become stale and a vendor bulletin in the retrieval index contains prompt-like text. The system marks feed freshness, excludes the untrusted instruction, and states that its confidence is degraded. It presents two source-cited hypotheses and the missing evidence for each. The operator follows the existing switching and escalation procedure.

If the assistant fails, the control room retains its alarm displays, procedures, radio, and staffing. Its outage cannot block the essential service. If it produces repeated unsupported recommendations, a circuit breaker disables the assistant while preserving logs.

The utility tests the system in a cyber-physical environment before production. The U.S. Department of Energy’s July 2026 announcement of the Stormbreaker testbed is official program evidence that sector-specific LLM and agent evaluation in power and OT environments is becoming an active capability. It does not validate any vendor model or utility deployment by itself.

Data and model assurance

Data contracts

For every sensor, log, forecast, document, and human report, record ownership, time synchronization, unit, calibration, sampling, latency, expected range, quality flag, retention, and use restriction. Missingness can be informative, malicious, or routine; the model must not silently impute a safe state.

Distribution and extremes

Split evaluation by season, geography, asset age, operating state, supplier, weather, communications loss, and event severity. Historical averages underrepresent black-swan combinations. Add simulation, digital twins, engineering scenarios, and fault injection while distinguishing synthetic evidence from observed history.

Model qualification

Version the model, prompt, tools, retrieval corpus, thresholds, and runtime. Evaluate calibration, false-alarm burden, missed critical events, robustness to corrupted inputs, and performance under load. A general model benchmark cannot establish fitness for a water plant or rail network.

Independent checks

Cross-check critical values with deterministic calculations or separate sensors. Validate units and physical bounds outside the model. Do not use the same model both to generate and approve a plan.

Change control

Provider update, new model, prompt edit, data remapping, retriever change, or added tool is a system change. Define whether it requires regression testing, safety review, regulator notice, operator training, or rollback preparation.

Cybersecurity for AI in OT and ICS

AI adds components and attack paths: model and data supply chains, APIs, vector stores, prompts, plugins, inference servers, training pipelines, and remote support. Apply zero trust and OT segmentation.

keep internet-facing and enterprise AI away from direct control paths;
use one-way or brokered data flows where appropriate;
isolate credentials by tool and environment;
allow-list actions and validate parameters;
treat documents, tickets, and sensor text as untrusted;
sign models, code, and configuration; verify at deployment;
log retrieval, prompts, tool calls, policy decisions, and human overrides;
test prompt injection, poisoning, evasion, model extraction, and denial of service;
preserve manual and deterministic fallback;
include the AI supply chain in incident exercises.

CISA and international partners’ principles of OT cybersecurity remain relevant: safe and secure OT requires architecture and lifecycle discipline. AI does not replace those foundations. Our cyber-physical security guide covers this intersection in more depth.

Human factors and operational authority

Operators need to know what the AI saw, what it did not see, how fresh the inputs are, which sources support a recommendation, and what action remains theirs. Avoid interfaces that show a single confidence score without consequences or alternatives.

Test:

alarm volume and prioritization under surge;
automation bias and reluctance to override;
handoff between shifts and teams;
use during fatigue, stress, and degraded communications;
accessibility, language, and local terminology;
recovery after the AI is disabled;
whether staff retain manual competence.

An override is not automatically an operator error. It may expose bad model context, an undocumented local condition, or an outdated procedure. Review overrides as safety evidence. Maintain stop-work authority and a non-punitive reporting channel.

Supplier and dependency risk

Map model provider, cloud, chips, libraries, data suppliers, integrators, remote administrators, and support channels. Contracts should provide change notice, incident reporting, audit evidence, vulnerability handling, data-use limits, service continuity, exportable logs, and exit assistance.

Do not accept “proprietary model” as a reason to skip system-level testing. You may not need weights to test outputs, tools, failure behavior, data paths, updates, and recovery. Apply vendor-risk procurement before the system reaches OT.

Metrics that matter

Measure service and risk, not only model accuracy:

severity-weighted missed events and false alarms;
detection-to-operator and operator-to-action latency;
calibration and uncertainty behavior;
stale, missing, or conflicting data correctly surfaced;
unsupported recommendation and unsafe tool-attempt rate;
operator acceptance, correction, override, and workload;
system availability and mean time to safe degradation;
fallback success and time to manual operation;
cyber incidents, blocked attacks, and time to containment;
model/data drift and change-regression findings;
restoration time, service continuity, and affected users;
performance by site, asset, environment, and population served.

Near misses and qualitative operator reports are leading indicators. A rare catastrophic hazard cannot be managed by waiting for enough production incidents to reach statistical significance.

Release gates

Before production, require:

documented intended use, prohibited use, authority, and safety owner;
hazard analysis tied to service-level consequences;
non-AI alternative and manual fallback;
data lineage, quality monitoring, and time synchronization;
sector-specific TEVV under normal, extreme, degraded, and adversarial conditions;
independent interlocks and physical bounds for any executable action;
OT security review, threat model, and supplier assessment;
operator training, usability tests, and demonstrated override;
incident, communications, recovery, rollback, and decommissioning plans;
approval by operations, engineering, safety, cyber, legal, risk, and appropriate regulators.

Use a staged path: offline replay, simulation, shadow mode, advisory pilot, limited sites, and only then broader operation. Each stage has quantitative thresholds and a stop rule. The operational readiness checklist can structure this evidence.

Incident response and recovery

Define AI-specific triggers: anomalous action, systematic wrong recommendation, drift, poisoned data, compromised credential, provider outage, unexplained model change, or loss of audit logging.

The response should:

place the system in a known safe or advisory-only state;
preserve volatile evidence and versions;
revoke credentials and isolate affected paths;
notify operational and safety leadership;
assess physical and public impact;
switch to practiced fallback;
communicate with regulators, partners, and communities as required;
validate remediation in a test environment;
reauthorize rather than automatically restart.

Run exercises that combine cyber, physical, supplier, and communications failure. Critical infrastructure often depends on other sectors; a power, telecom, cloud, or fuel disruption can cascade.

Frequently asked questions

Can a high-accuracy model control critical infrastructure?

Accuracy alone is insufficient. Control requires hazard analysis, deterministic constraints, independent protection, rigorous validation, regulatory compliance, and safe fallback.

Is on-premises deployment safer?

It may reduce some network and data risks while increasing patching, physical, insider, and operating risks. Choose from the threat model, not a slogan.

What is the best first use case?

Read-only analysis with measurable value, source-cited evidence, no control permission, and an existing manual process—such as log correlation or maintenance-document retrieval.

How often should the system be revalidated?

Continuously monitor and trigger formal revalidation after material model, data, prompt, tool, environment, or operating changes. Periodic review alone is not enough.

Who can stop the AI system?

Named operators and safety owners must have an immediate, tested, and logged stop mechanism that does not require the AI itself to cooperate.

The 2026 operating principle

Critical-infrastructure AI earns trust through bounded authority and tested recovery. It must reveal degraded evidence, coexist with independent protection, resist cyber and data failure, respect operator authority, and return the service to a known safe state. The objective is not maximum autonomy; it is resilient public service under normal conditions and the worst credible day.

Source notes

Sources reviewed and current as of July 30, 2026: