An operating model for production AI security

At a glance

Start with a consequential workflow and trace its models, data, credentials, decisions, and downstream effects.
Use identity and integrity controls alongside authorization, behavioral evaluation, human oversight, and tested containment.
Collect evidence during normal operation, then map it to the obligations and assurance questions that apply.

Start with the action the system can take

An AI assistant becomes a different security problem when it can issue a refund, change a production configuration, or recommend a consequential decision. The operating question expands from whether the answer sounds reasonable to whether the system had permission, used the expected components, and left enough evidence to explain what happened.

My starting point is one workflow, one accountable owner, and one concrete failure. Map the human request through retrieval, model inference, tool calls, and the final application. Record where data crosses a boundary and where an action becomes difficult to reverse. That map sets the control priorities.

Separate risk management from legal applicability

NIST's AI Risk Management Framework provides the voluntary Govern, Map, Measure, and Manage functions. Its Generative AI Profile adds guidance for generative systems. Use them to organize responsibilities and evaluation, with controls proportionate to the use case. Framework adoption alone does not establish regulatory conformity.

The EU AI Act has different obligations for different roles and uses. Following the AI Omnibus that entered into force on 27 July 2026, covered Annex III high-risk rules apply from 2 December 2027; covered Annex I product high-risk rules apply from 2 August 2028. Article 50 transparency obligations began applying on 2 August 2026, with a specific 2 December 2026 transition for certain existing synthetic-content systems. Check classification, role, exceptions, and transitional provisions before assigning a deadline.

Reusable technical evidence can support several assessments. Each assessment still needs its own requirements mapping, including data quality, human oversight, performance, privacy, and product-specific obligations.

Find the gaps between individual tools

The system needs defenses against attacks on AI and misuse through AI. A poisoned retrieval document may redirect a tool call. A substituted model artifact may bypass release review. Synthetic media may impersonate a colleague. Different failures need different controls; signing everything cannot resolve all three.

A model registry, prompt filter, and agent dashboard each answer part of the problem. The gap appears when no shared identifier connects the approved release to the running workload, its delegated permissions, and the resulting action. Treat retrieved documents and tool responses as untrusted input, enforce permissions outside the model, and test the complete workflow. OWASP's prompt-injection guidance describes why model behavior needs this surrounding control structure.

Six capabilities to connect

These capabilities can span several systems. The important integration is a verifiable handoff between them. For a hosted model whose weights you cannot inspect, record the supplier's model identifier, configuration, contractual evidence, and change notifications. State the resulting limits on independent verification.

Capability	Operating control	Evidence to retain
Model and release integrity	Identify the approved model, application, prompt configuration, and tool versions. Verify artifact digests and trusted signatures where artifacts are available.	Release identifier, signer, digest, evaluation results, approval, and deployment record.
Agent identity and delegation	Give the workload an attributable identity. Bind each task to its initiating person or service and narrowly scoped authority.	Workload identity, delegating principal, task, audience, scope, and expiry.
Data and retrieval governance	Track origin, access permissions, transformations, retention, and evaluation of training or retrieval data. Preserve known gaps in supplier evidence.	Dataset or document versions, access decisions, processing history, and quality review.
Runtime policy and boundaries	Authorize tool operations at the gateway and destination. Limit network egress, data access, execution privileges, and consequential actions.	Policy version, decision, approval, tool parameters, destination, and result.
Output provenance	Link important outputs to their producing workflow. Use Content Credentials for supported media workflows and suitable application records elsewhere.	Output identifier or digest, producing service, release, time, and provenance validation result.
Evidence and interoperability	Join records with stable identifiers and export them in documented formats. Protect logs against alteration and unauthorized access.	A reconstructable event timeline, retention policy, verification procedure, and export.

Follow one request through the controls

Register the task: identify the user or service, business purpose, allowed resources, and accountable owner.
Authenticate the workload: validate its identity and issue only the credentials required for this task. SPIFFE is one open approach to workload identity.
Authorize the operation: evaluate identity, delegation, resource, action, and current policy. A valid identity alone supplies no permission.
Check the boundary: validate tool arguments and destination, apply data restrictions, and require specific human approval when the action crosses a defined consequence threshold.
Record the outcome: connect the decision, model release, relevant input references, output, and downstream result without placing credentials or unnecessary personal data in logs.
Contain and recover: stop new execution, block access, cancel queued work where possible, and preserve evidence. Validate the affected downstream state before resuming.

Be precise about what the proof establishes

A model signature supports artifact integrity and attribution to a trusted signer. It does not establish model safety, accuracy, fairness, or resistance to prompt injection. A signed dataset manifest protects a claim about the dataset; it does not establish that collection was lawful or the data representative.

C2PA Content Credentials provide verifiable content provenance. They do not prove that a depicted event happened or an AI answer is true. Provenance may be incomplete or removed, and missing credentials alone do not establish deception. Test the actual export and distribution path with the intended verifier.

Remote attestation can support a decision about a measured execution environment. RFC 9334 describes the roles of attester, verifier, and relying party; it is an architecture, not a universal certification. Define accepted measurements, reference values, freshness, and failure behavior before using attestation to grant access.

Assess readiness through demonstrated behavior

Score workflows separately. A well-controlled assistant does not establish readiness for an autonomous payment agent. Use the weakest consequential control to set the next priority.

Stage	What the team can demonstrate
Experimental	A working feature, but uncertain ownership, credentials, release history, or incident response.
Catalogued	A maintained inventory of workflows, models, suppliers, sensitive data, owners, and risk classifications.
Controlled	Verified releases, attributable agents, scoped permissions, and an enforced boundary on a consequential workflow.
Evidenced	Connected records, maintained evaluations, tested containment, and an incident timeline produced from retained evidence.
Sustained	New releases trigger evaluation; changes and exceptions have owners; evidence and recovery controls are exercised regularly.

Build a first operating cycle

This is a suggested implementation cycle. Existing exposure may require immediate containment. Start a customer-support workflow with customer-data access and refund limits; start industrial maintenance with the handoff from recommendation to equipment command; start clinical decision support with evaluation and human review. These are entry-point examples, not reported customer outcomes.

Track inventory coverage, excessive privileges, failed policy checks, evaluation regressions, containment time, and missing evidence. Keep the denominator visible so a better-looking dashboard cannot hide new unmanaged workloads.

Days 1–30: inventory one production workflow and adjacent employee-adopted tools. Classify data and authority. Name owners, define unacceptable outcomes, and remove unnecessary privileges.
Days 31–60: connect release records, workload identity, policy decisions, and outcome logs. Add boundary checks and approval for consequential actions. Test unauthorized requests, hostile retrieved content, and supplier or model changes.
Days 61–90: run a containment exercise and reconstruct its timeline. Export the evidence for an independent reviewer. Resolve gaps, document residual risk, and choose the next workflow.

Questions to take back to your team

Can we verify which model, application configuration, and tools produced a sampled production action?
Can we distinguish the agent from the person or service that delegated its authority?
Where is each resource and action permission enforced, including paths that bypass the gateway?
Which training or retrieval provenance claims can we verify, and which depend on a supplier?
What changes trigger fresh behavioral evaluation and release approval?
How long does containment take across running sessions, delegated credentials, queues, and downstream systems?
Can an independent reviewer reconstruct an incident and map the evidence to the specific applicable requirements?

Sources & further reading

NIST AI Risk Management FrameworkVoluntary risk-management framework; supports the obligations and readiness discussion.
NIST Generative AI ProfileCompanion guidance for generative AI risk management and evaluation.
European Commission AI Act implementation timelineStaged application and the specific Article 50(2) transition, reviewed 5 September 2026.
European Commission: AI Omnibus enters into forceAmended high-risk timetable and entry into force on 27 July 2026.
OWASP: Prompt InjectionPrompt-injection risks and surrounding security controls.
SPIFFE overviewOpen workload-identity specifications and short-lived identity documents.
C2PA Content Credentials explainerProvenance, verification, incomplete history, and limits on establishing truth.
IETF RFC 9334: RATS ArchitectureRemote-attestation roles, evidence appraisal, and freshness considerations.

A personal field guide. The checklists and operating frameworks are my recommendations; the linked authorities define their own requirements and scope. Review product and jurisdiction-specific obligations against the current source material.