For most of 2024 and the first half of 2025, AI governance for agents was a tomorrow problem. By mid-2025 it had become a this-quarter problem. The EU AI Act began entering force in stages, the NIST Generative AI Profile shipped, and procurement teams started asking pointed questions in vendor diligence calls. This piece is the working map of the landscape and the controls a serious team would put in place to be ready for it.

This is not a legal document. Treat it as an engineering and operations playbook: what to build, what to log, what to write down. For specific regulatory advice, consult counsel.

Why governance moved up the agenda in 2026

Three forces converged. First, the EU AI Act passed in March 2024 and entered into force in August 2024 with staggered applicability through 2027 (European Commission). Second, the NIST AI RMF added a Generative AI Profile in July 2024 (NIST AI 600-1). Third, enterprise procurement adapted: vendor diligence questionnaires from 2025 onward routinely include questions about training data lineage, evaluation rigour, audit trails, and human oversight.

The implication for agent builders is that governance is now a feature gating enterprise sales, not a back-office concern.

EU AI Act and the agent question

The Act categorises systems by risk. Prohibited systems (social scoring, exploitative manipulation) cannot be deployed at all. High-risk systems (employment decisions, credit scoring, critical infrastructure, certain healthcare and education uses, biometric categorisation) require risk management, data governance, transparency, human oversight, and conformity assessment. Limited-risk systems require transparency obligations. Minimal-risk systems are unregulated.

Where agents land

An agent itself is not automatically high-risk; the use case decides. An agent that filters job applications, evaluates creditworthiness, or makes healthcare recommendations falls into high-risk categories. An agent that triages internal inboxes generally does not.

Applicability timeline

(European Commission AI Act page)

NIST AI RMF and the Generative AI Profile

NIST AI RMF 1.0 (January 2023) is a voluntary US framework with four functions: Govern (organisational policy), Map (context and risk identification), Measure (quantification), Manage (response and continual improvement) (NIST AI RMF).

The Generative AI Profile (NIST AI 600-1, July 2024) extends the RMF with generative-specific risks: confabulation, data privacy, harmful bias, intellectual property concerns, information integrity, and value chain risk (NIST AI 600-1). The profile is the right baseline for agent risk programs in the US.

SOC 2 mapping for agent platforms

SOC 2 is the AICPA Trust Services Criteria framework that B2B SaaS procurement teams treat as table-stakes. Security is required; Availability, Confidentiality, Processing Integrity, and Privacy are optional and chosen based on what the platform claims (AICPA SOC for service organizations).

Agent-specific overlays

Timeline reality

SOC 2 Type II requires 6-12 months of operating evidence post-Type I. Total time from "we should do SOC 2" to "we have a Type II report" is typically 12-18 months. Start before the procurement conversation, not after.

Model cards and capability cards

Model cards (Mitchell et al., 2019) are short structured documents describing a model's intended use, training data, evaluation metrics, known limitations, and recommended uses (Mitchell et al., Model Cards for Model Reporting, FAT* 2019). They are the primitive that ties technical reality to compliance documentation.

For agents the model card extends with a capability card: tool inventory, blast radius bounds, retry budget, audit retention, human-in-the-loop trigger conditions. The capability card is the artefact a buyer reads during procurement; it should answer "what can this agent do, what can it not do, what happens when it fails".

Acceptable use policy

An AUP for agents codifies the operational envelope. Sections to include:

The AUP is reviewed by Legal, Security, and the product team together. Sign-off is logged.

Data governance and residency

Three primitives to get right.

Data lineage

For every dataset the agent touches: source, owner, classification (public, internal, confidential, restricted), retention, downstream consumers. This is the data sheet (Gebru et al., Datasheets for Datasets, 2018).

Residency

Where data sits at rest, where it transits, where it is processed. EU customer data should stay EU-resident; this implies provider region selection at the LLM and storage layers.

Deletion and the right to erasure

Customer data deletion requests propagate to agent memory, audit trails (where legally permissible), and any derived embeddings. The right to erasure under GDPR Article 17 applies to AI systems the same as any other data processor.

Human oversight requirements

EU AI Act Article 14 specifically requires effective human oversight of high-risk AI systems, including the ability to "fully understand the capacities and limitations" of the system and to "decide not to use the high-risk AI system or otherwise disregard, override or reverse the output" (EU AI Act Article 14, AI Act explorer).

For agents the practical implications are: every high-risk decision must have a human-reviewable trail, a stop-the-action mechanism, and a path to reverse the decision. The kill switch (see AI agent security best practices) is the regulatory backstop here, not an optional feature.

Frequently asked questions

Does the EU AI Act apply to AI agents?

Yes when the use case is regulated. High-risk applications (employment, credit, healthcare, biometric categorisation) require risk management, data governance, transparency, human oversight. Phased applicability from 2025 through 2027.

What is the NIST AI Risk Management Framework?

Voluntary US framework with four functions (Govern, Map, Measure, Manage). The Generative AI Profile (AI 600-1) extends it for generative systems.

How does SOC 2 apply to an AI agent platform?

Same Trust Services Criteria as any SaaS. Agent-specific overlays include audit trails (CC6, CC7) and change management (CC8). Type II requires 6-12 months of operating evidence.

What is a model card for an AI agent?

A structured document describing intended use, training data, evaluation, limitations. For agents, extended with tool inventory, blast radius, and retention.

What is an acceptable use policy for AI agents?

Defines permitted and prohibited use, data handling, escalation, customer disclosure, and review cadence. Contract between platform team, business unit, and compliance.

Three things to ship this quarter

Sources