Control that holds when the model drifts
How governance and engineering leaders are handling AI that changes behaviour after approval, and where human sign-off stopped scaling.
In this recap
- The two assertions we opened with
- Whether tested still means released
- Human in the loop, and where accountability sits
- Controls by design, boundaries and falsifiability
- Third-party AI, and intake built for static products
- Buy, build, orchestrate, and the shadow at the desk
- Where the room did not agree
The room
Ten senior practitioners, one morning, one table. Responsible AI leads, enterprise architects, quality engineering heads, compliance managers and engineering managers, drawn from banking, insurance, payments, retail, development finance and the public sector.
The session ran under Chatham House convention. Everything below is reported without names or organisations, which is the condition that lets people say what they actually think rather than what their communications team would prefer.
This was the first room in the AI Governance track, split out from the software supply chain series after the July session asked repeatedly for the two topics to be separated.
The two assertions we opened with
The facilitator put two propositions to the room and asked people to agree or disagree. Before you read what happened, decide where you stand.
Your change control process assumes the thing you tested is the thing that runs. That assumption is now false and nobody has rewritten the process.
Some held that this describes an old process failure now under load rather than anything new. Others argued the artefact itself has stopped being stable. Both readings are set out in the next section.
The AI in your estate did not arrive through an AI approval process. It arrived inside software you bought, and your third-party intake is built for products that do not change behaviour after purchase.
The examples came fast and they were consistent across sectors. One participant dissented, on the grounds that contract-scoped engagement forces a review whenever a supplier changes what it delivers.
Whether tested still means released
Several people rejected the first assertion outright, and the pushback was not defensive. One put it as an ambition to go faster being paid for by letting go of the practices that make speed safe. The room divided roughly along these lines.
The assertion does not hold
If what you tested is not what you release, something has already gone wrong upstream. The cause is missing engineering practice, not a new property of AI.
Human validation and verification is precisely the thing that confirms test equals release. Rogue agent behaviour, which happens more often than anyone would like, points at a broken process rather than an invalid assumption.
The assertion holds
The artefact itself is no longer stable. Models drift between versions. Harnesses change behind an unchanged API.
One participant described the same prompt returning a materially different answer from the same named model ten days apart. Fitness for purpose, on that reading, is never permanent.
Underneath both positions sat a shared observation: the volume of generated code has outrun the review capacity that change control quietly depends on.
Human in the loop, and where accountability actually sits
This was the longest thread of the morning, and the one where the room came closest to agreement.
Almost everyone accepted that human review does not scale. Pull request review is already the bottleneck in most estates represented. A human signing off work they cannot realistically read is a rubber stamp, and a rubber stamp defeats the purpose of the control.
Two counterweights were pressed hard.
Why oversight fails
Volume is too high, and people are biased and fatigue. One participant argued the industry has been sold an illusion that human in the loop will persist, citing the documented harm done to early social media content moderators.
Why it cannot be removed
Regulation will not permit the human to disappear, because someone has to be accountable when an automated decision goes wrong. Automated decision-making law in several jurisdictions already fixes accountability on a person, not a system.
A third response was a reframe rather than a rebuttal: the answer is not removing the human but moving them. From in the loop to on the loop. A human will not review hundreds of individual agent decisions, so they move up to the end-to-end system, with an explicit threshold below which the agent acts alone. The worked example given was a refund process where a small value is handled autonomously and a materially larger one pulls a person in, with the threshold itself as the guard rail, moving further along the chain as confidence in the system grows.
That reframe carried an auditability requirement with it. Given two agents that reach the same answer, one using the tools available to it and one improvising because the tools were absent, the human at the end has to be able to tell which is which. That is a design-time requirement, not something you add afterwards.
Controls by design, boundaries and falsifiability
A recurring argument, and one the room broadly treated as unglamorous rather than novel: controls have to be designed into the process, not layered on after it. Anyone who has worked in a regulated environment recognised it immediately, because it has always been true of any control.
Where it got interesting was in what a control means when the thing being controlled is stochastic. Several strands ran through it. Whether an agent can detect a boundary that a human previously judged by instinct. Whether a process is robust enough that several competent people would reach the same judgement, because if it is not, agents will not save it. Whether the output of a system is falsifiable and whether the evidence supports it. And whether the audit trail can show an assessor enough of the decision path without exposing the underlying data.
Systems engineering has required a testable requirement for decades. Most requirements now presented are wishes.
A position argued in the room, reported without attribution.
One participant made the case for starting from published standards work and building a comprehensive requirement set across architecture, platform, application, model and agentic layers, then deriving boundary conditions for each and defining what a red team result has to demonstrate before it counts as a pass. Standards came up repeatedly through the morning, always as the starting point and never as the finish line.
Modularity was raised as a design answer in its own right. A single process doing everything end to end is a poor design, whereas splitting it means an unvalidated output at one stage can be caught by a control at the next.
Third-party AI, and intake built for static products
The second assertion drew the strongest recognition in the room, and the examples came quickly.
Onboarding, in most of the organisations represented, is long and thorough. It also validates a product as it exists on the day. Participants described AI features switched on by default by existing suppliers, upgrade cadences on multi-year contracts that cannot be declined, features flagged a few weeks ahead with no realistic option but to test and re-engineer around them, and foundation models changing underneath an unchanged contract.
A related difficulty is classification. There is a real governance difference between a pre-approved foundation model and a bespoke model a supplier has built themselves, and between software that operates on your data and software built using AI that assists without consuming it. Several people said the treatment should differ, and that their process does not yet reliably tell them which one they are buying.
The responses discussed were consistent across sectors: a central inventory holding third-party AI on the same footing as internal AI, approval with appropriate specificity rather than a single questionnaire, ongoing monitoring of actual behaviour rather than a point-in-time attestation, and reassessment triggered whenever an assessment input changes.
One dissent is worth recording. A participant argued their organisation would not experience the assertion as stated, because engagement is contract-scoped: a supplier wanting to add an AI feature has to either ask for more money or replace a component, and either one triggers review. The caveat he added is the indirect case, where a supplier providing research or screening output uses AI to get there and the provenance of the answer is the thing you actually need to control.
Buy, build, orchestrate, and the shadow at the desk
The last thread was about where the boundary of the governed estate now sits, and the honest answer was that nobody in the room was confident about it.
Procurement thresholds let a team buy AI-containing software without full review. Teams build internal tools overnight that never go near production, and the prototype cycle has collapsed from months to a single evening. Sandbox and playground environments carry data exposure identical to production even though the process around them is not. Generated dashboards and documents would once have been ordinary desktop output and now look a lot more like software.
Set against that, several described a deliberate move from buying towards building, driven partly by capability and partly by supplier costs rising on the back of infrastructure scarcity. One framed the choice as no longer buy or build but buy, build or orchestrate, because a finished product is rarely what anyone wants and organisations increasingly assemble models, data sources and components themselves.
Not everything built needs to reach production. Heavy governance over experimentation risks telling capable teams not to innovate.
A position argued in the room, reported without attribution.
The balance suggested was explicit accountability for experimentation, held by a named manager, rather than another process gate.
Where the room did not agree
Worth stating plainly, because the disagreement was the most useful part of the morning.
Whether the change control problem is new, or an old process failure now under load.
Whether human oversight is a durable control or a failing one, and whether the volume argument holds in sectors where the backlog is already the crisis. Public sector waiting lists were offered as a counterexample, alongside diagnostic imaging as a case where machine review is genuinely clearing a backlog because there are not enough specialists.
Whether cost is a token problem or an ecosystem problem. The stronger argument was the second: security, data governance, tooling, and people who know how to use it.
Whether governance work is substantive, or largely about making ourselves feel better while the real work is understanding the data and where the data loops are.