Governing Generative AI: Managing Risk Across the Model Lifecycle

Why Existing Model Governance Does Not Transfer

Organisations with mature model risk management — banks in particular — often assume their existing framework extends to generative AI. It partly does, and the parts that do not are the parts that matter.

Assumption in classical model governance	What generative AI does instead
The same input produces the same output	Non-deterministic by design; validation must be statistical, not exact
Inputs are structured and bounded	Inputs are free text from users, documents and other systems
The training data is known and controlled	Foundation model training data is largely opaque to the deployer
The model is revalidated on a schedule	The provider may update the model underneath you
Output is a score consumed by a process	Output is language consumed by a person, or an action taken on a system
The model computes; it does not act	Agents call tools, access systems and execute business transactions

The last row is the one that changes the risk profile most and receives the least attention in governance documentation written before 2025.

Provenance in a World of Opaque Base Models

You cannot audit the training corpus of a commercial foundation model. That is a fixed constraint, and the correct response is to govern rigorously the layers you do control.

The layers you own

• Retrieval corpora. Every document indexed into a vector store. This is the largest ungoverned surface in most enterprises today.
• Fine-tuning data. Where used, fully within your control and fully your liability.
• Prompt templates and system instructions. Versioned artefacts that materially determine behaviour, yet often live in application code with no change control.
• Tool and API surface. What the model can reach. The most direct control you have over what it can do.
• Output handling. What happens to what it produces.

The retrieval corpus problem

A vector store built from a document repository inherits every governance defect of that repository, and the embedding process typically strips the context that would have flagged them. Superseded drafts, documents with restricted circulation, content past its retention date and material subject to legal hold all become equally retrievable.

The remedy is to carry governance attributes through the pipeline and filter at query time:

• Every chunk retains a pointer to its source document, classification, last-modified date, retention status and permitted audience.
• Retrieval filters on the requesting user’s entitlements, not just on semantic similarity.
• Re-indexing is triggered by source changes, so deletions and reclassifications propagate rather than persisting in the index indefinitely.
• Citations returned to the user resolve to the actual source, so a human can verify.

Prompts and Outputs as Governed Artefacts

Prompts are configuration, not code comments

A system prompt determines model behaviour as decisively as any parameter. Yet in most deployments it sits in a source file, edited without review, with no record of what changed or why.

Treat prompt templates as versioned, reviewed artefacts. At minimum: version identifiers recorded with every inference, change approval proportionate to the system’s authority tier, and the ability to answer “which prompt version was in force on this date?”

Output logging that survives an audit

The EU AI Act requires automatic logging for high-risk systems. The practical standard to design for is reconstruction: eighteen months later, for a specific interaction, can you produce the model and version, the prompt template version, the user input, the retrieved context, the output, any tool calls made, the human decision that followed, and the policy in force?

Two tensions have to be resolved deliberately. Logging user inputs may capture personal data, so retention and access controls on the log store need to be at least as strict as on the source data. And full logging of high-volume systems is expensive, so sampling is legitimate for low-tier systems provided the sampling rate is documented and the criteria for full capture — flagged outputs, overrides, escalations — are defined.

Evaluation as a standing process

Point-in-time validation does not hold when the provider can update the model beneath you. Maintain a golden set of representative inputs with expected characteristics, run it on a schedule and on every model or prompt change, and alert on drift. This is the generative equivalent of monitoring, and it is not optional for anything above Tier 1.

The Agentic Shift

Enterprises are no longer deploying AI that only generates answers. They are deploying agents that discover information, call tools, access systems and execute business actions. Agent registries, agent identities and governed agent catalogs are becoming core components of the enterprise AI control plane.

But an agent catalog has to be more than an inventory. For every agent, the organisation should be able to answer:

• Who owns and sponsors it?
• What business purpose does it serve?
• Which data, tools and systems can it access?
• What decisions or actions can it execute?
• Where is human approval required?
• How is its behaviour monitored and audited?

Authority tiers

The governance model should recognise a critical distinction: an agent that assists a person does not carry the same risk as an agent that executes changes in a system of record. We recommend classifying every agent explicitly:

Tier	Capability	Required controls
0	Reads and summarises only	Registration, data access controls, sampled logging
1	Drafts and proposes; human sends	Full logging, quality sampling, named owner
2	Executes reversible changes	Audit trail, monitored thresholds, tested kill switch, rollback procedure
3	Executes consequential or irreversible actions	Mandatory approval gate, dual control on config, board-visible reporting

Agent identity is a real requirement

Agents acting under a shared service account are ungovernable — you cannot attribute an action, scope permissions meaningfully or revoke access without collateral damage. Each agent needs its own identity, its own least-privilege permission set, and credentials that can be rotated and revoked independently.

Prompt injection as an access control problem

An agent that reads untrusted content — email, web pages, uploaded documents, ticket text — will eventually encounter instructions embedded in that content. Treat every input the agent did not receive directly from an authenticated user as data, never as instruction. The durable defence is not better prompting; it is ensuring the agent’s permissions make the damaging action impossible regardless of what it is told.

An Operating Model That Scales

Governance that reviews every AI use case individually does not survive contact with demand. Three mechanisms keep it workable.

1. Pre-approved patterns. Publish reference architectures — retrieval over an approved corpus, a Tier 1 drafting assistant, an internal summarisation tool — that carry standing approval. Building to a pattern proceeds without review; departing from one goes to the council.
2. Risk-proportionate gates. Tier 0 self-registers. Tier 3 receives a full assessment. Applying uniform rigour teaches teams that governance is theatre.
3. Governance where the work happens. Registration in the repository or the catalog, not a form on an intranet page. Evaluation in CI. Policy checks in the deployment pipeline.

Incident response, rehearsed

Assume a generative system will at some point produce something harmful, leak something confidential or take an action it should not have. Decide in advance who can disable it, how fast, what gets communicated and to whom, and how you reconstruct what happened. Then rehearse it. An untested kill switch is a hypothesis.

The Principle Underneath

Generative AI governance is frequently framed as a technical challenge — better evaluation, better guardrails, better filtering. Those help. But the systems that get organisations into trouble are almost never the ones that were carefully evaluated. They are the ones nobody knew existed, or the ones whose permissions nobody scoped.

In the agentic era, AI governance must manage not only what intelligence knows — but also what intelligence is allowed to do.

Cataloguing agents creates visibility. Governing their authority creates trust. The organisations handling this well are not the ones with the most sophisticated evaluation harnesses. They are the ones who can name every agent in production, say who owns it, and show what it is permitted to touch.

Published by KRISID · 25 June 2026. This paper reflects our delivery experience and publicly available sources at the time of writing. It is general guidance, not legal advice — regulatory obligations vary by jurisdiction and by how a system is used.