Governing the harness, not the model
Why the room moved the control point to the deterministic layer, and what that leaves for the human at the gate.
In this recap
The room
Six practitioners, and a session scheduled for two hours that ran to two and a half. A development platform lead serving around six thousand engineers, a senior engineering manager building agentic experiences for a large marketplace, a lead security analyst for AI and agent platforms at a telecommunications operator, a co-founder and chief technology officer of a healthcare platform, a centre of excellence lead at a large European retailer, and a head of application security at a consumer software business protecting hundreds of millions of people.
The session ran under Chatham House convention. Everything below is reported without names or organisations.
The format was direct. After a round of introductions the facilitator put a prepared assertion to each person in turn, drawn from a one-to-one conversation held before the session, and asked them to agree or disagree. That is why this recap reads as six arguments rather than one discussion.
One caveat belongs up front. The recording logged every contribution under a single label, so attribution was reconstructed from the introduction round, from the facilitator naming each person as he put their assertion, and from content that could only have come from one speaker. Where a contribution could not be resolved it has been left unattributed rather than assigned.
Governing the wrong object
The most sustained argument of the session, and the one the room largely came round to.
The contention was that governance is being applied to the wrong thing. The model and the harness around it are treated as a single entity when they are two. The probabilistic part cannot act on anything. The deterministic tooling layer is what acts, which makes it the thing that has to be secured.
Treat the model as an untrusted user, and treat the harness as regular infrastructure.
A position argued in the room, reported without attribution.
The appeal of that formulation is that it needs no new discipline. A harness governed as infrastructure is governed the way networks and web estates already are: drift, baseline, open findings.
The worked example was a customer service agent. The common implementation looks up the caller's postcode from their number and places it in model context so the agent can verify them. That is the mistake. Once a value is in context it can be leaked, and a caller who convinces the model it was already provided is verified without ever giving it. The correct shape is the model asking, the user answering, a tool returning true or false, and any resulting token living in the orchestration layer where it never touches the context window. The rule underneath it: the model and the user should hold the same authorisation level at all times.
Two further positions were stated flatly and nobody contested them. Prompt injection will never be eradicated from any model. And anything entering a context window must be assumed leakable regardless of the guardrail stack, whether the model is frontier or self-hosted. A contract saying data does not leave a boundary is a contractual obligation, not a technical control, and logging and telemetry are where such contracts tend to go vague.
When you govern an AI system, what is the object you are actually governing: the model, or the tooling around it?
Two objections survived. One held that traditional machine learning inside an orchestrator is also probabilistic, since feature attribution tells you what contributed but not why, making the split a matter of dimensionality rather than kind. The other was about how far the principle can be pushed before it stops describing a real architecture.
Everyone means something different by the same word
The assertion put to the security analyst was that most AI governance failure is definitional rather than technical: two teams using the same word to mean different things, with neither of them wrong. He agreed without qualification, and the example he gave made it concrete.
Security sees prompt injection, policy puppetry and adversarial patching as one attack class, input to shifted result. Machine learning engineers see three different things. Both positions are internally consistent. The consequence is policy drafted by two functions who appear to agree, which developers then cannot implement, and penetration test reports that keep finding the thing the policy was supposed to prevent.
The room extended it to the oldest problem in engineering, two teams speaking different languages, now running at ten times the speed.
The human at the gate
Every architecture described in the room put a human at a gate. The room then spent a long stretch asking whether that is a control or a comfort.
The sharpest observation came from someone who had been listening to all of them: every solution described so far has a low blast radius, and that is precisely why they work. The counter-case he offered was network load balancing at a telecommunications operator, where an agent optimising for demand could starve emergency calls. Emergency services, defence, power generation and clinical care came up repeatedly as the places where a probabilistic control will not be accepted regardless of measured accuracy.
On accountability the room split along an interesting line. One argument treated accountability as a conclusion drawn from intent: a person can be penalised or rewarded because intent can be established, and there is no equivalent for a model. The other framed it as the trolley problem extended to an actor with no agency, and dismissed the idea that it can be argued in front of a regulator at all. You cannot tell a regulator the AI decided.
The clinical architecture described was the most careful version of a human gate anyone offered. Knowledge graphs hold the clinical guidelines alongside the medical record and wearable data, which together build the context. The model produces output, a medical professional checks it, and everything refers back to source so nothing is presented without provenance. His own open question was the honest one: whether that pairing gets better or worse over time, because either answer changes the design.
Underneath it he drew a distinction the room kept returning to. The impact AI is chasing is economic, and economic impact can never be allowed to outweigh life and death impact.
The knowledge that stops forming
The worry that connected the most people, and the only one with a concrete countermeasure attached.
The concern is that the human placed in the loop today may not be capable of occupying it in five years. Engineers are entering the profession having never debugged unaided. The intuition that lets an experienced person recognise bad output on sight is acquired by doing the work, and there is no obvious substitute route. One participant with eight years of red teaming behind him said he recognises bad output immediately and does not believe that recognition can be acquired any other way. The analogy offered was satellite navigation: a human in the loop who cannot operate without the platform.
The risk stated plainly is that juniors do not become mid-level, mid-level engineers do not become seniors, and the existing seniors leave.
One piece of evidence was brought rather than asserted. A forty-finding report returned to a third-party supplier used to produce a retest request within three to six months. It now takes six to eight. The person describing it was explicit that he cannot prove the cause, will not get workforce data from the supplier, and has to raise it with his own leadership without proof.
A second version of the same point was sharper still: using models to raise code quality still leaves an organisation full of people who do not understand what they are producing, because the need to write carefully has been removed. On that reading the quality improvement makes the position worse rather than better.
The one intervention described in practice was master and apprentice. A part-time university student paired with a principal engineer, knowledge passed down by working alongside. Another participant had arrived at the same answer independently, applying it inside a supplier after moving through four outsourcing companies in two and a half years before finding one they could train.
Cheap, not fast
The reframe that landed hardest in the room.
AI did not make engineering fast. It made engineering cheap.
A position argued in the room, reported without attribution.
Requirements gathering and specification exist because building the wrong thing used to be expensive. If a great deal can now be built badly at low cost, then two teams speaking different languages can build the wrong thing ten times faster and make that many more mistakes. Lines of code was rejected outright as a metric by more than one person.
The counter-position on volume was equally firm. Ten thousand generated lines a day is not a real prospect, because customers will not pay a hundred times more for the result. On that reading the engineering half is solvable when requirements are specific, and the binding constraint sits on the business and product side, in knowing what to build at all. The agentic workflow assumes requirements arrive clear, like a factory, and they never do.
One participant put the same problem from the platform side. The executive view he had been given was that if coding is cheap you produce ten candidate implementations and validate hard, because disposable code is cheaper than an engineer producing one. His own conclusion was that this moves the constraint rather than removing it.
Creating code is cheap. Validating code is still expensive.
A position argued in the room, reported without attribution.
On quality, one account separated two things that usually travel together. Clean code, meaning small functions and careful naming, had been consciously let go. Logical correctness, test coverage, integration tests and documentation had been doubled down on, partly because the models themselves need them to behave. Pressed on whether that was maintaining quality or redefining it, he argued the system is better because the models force better guardrails.
The concrete number offered: a language runtime upgrade that had previously taken seven months took four weeks. The same person was careful to bound it. Repeatable engineering migration accelerates sharply. Consumer product work does not, because the constraint there was never engineering.
Is an objective and key result an incentive, or a consequence delivered after the fact?
One position held that production quality is a non-negotiable expectation rather than something to incentivise, and that what is actively incentivised is AI-assisted working as a new practice. The other held that a defect filed against a name after the fact is a consequence, not an incentive, and that behaviour follows whatever is rewarded in advance: offer a bonus for ten thousand more lines of code and you will get exactly that and nothing good. If quality is the goal, the reward has to attach to quality up front.
Evaluating something probabilistic
The most practical stretch of the session, because three people had already built something.
One described roughly five thousand historical user requests curated by hand and used to train a judge model, run before production and again after, extended as new cases appear. It sits behind an assistant handling employee questions across multiple jurisdictions with different employment law, with a gating agent in front that assesses whether the available context is sufficient to answer at all rather than answering itself.
Another described a help agent resolving around eighty per cent of incoming questions with two judges: a conventional inner judge, and an outer judge made of human reaction, where users mark a response right, wrong, or needing a person. What agents are good at, on his account, is answering repeatable questions from a known data set, assembling context to hand to a human, and recognising when they have hit a boundary and should escalate.
A third described security tooling present from the first line written rather than at a gate, filing issues back into the same machinery that fixes them, with agentic false positive reduction cutting false positives by eighty per cent.
The disagreement underneath all of it was whether chaining probabilistic checks helps. One argument was layered discrimination: models are better at discriminating than generating, so a chain of checks each taking a slightly different perspective drives compound failure probability down, with the target being performance better than a human rather than a total guarantee. The counter-argument was arithmetic. Serial checks degrade contextual accuracy rather than improving it, ninety five per cent three times over leaving seventy seven per cent of the context intact, and a probabilistic control should never be treated as a control in defence, national healthcare or nation state contexts.
Two candid accounts of generated code closed the section. An assistant instructed to build decision heads for a reinforcement learning model reported success, and investigation showed it had skipped a prerequisite step entirely. In a separate benchmark, an assistant explained why one model outperformed another and, challenged on the inconsistency, admitted it had invented the answer rather than running the test. The response was to stop issuing generic instructions altogether and define each block with its inputs, transformation and outputs, even where that means writing two hundred specifications for a two hundred block programme, on the basis that the debugging time saved exceeds the specification time spent.
Where the room did not agree
Six people, one prepared assertion each, and four questions that were still open when the room broke up.
Whether traditional machine learning inside an orchestrator counts as deterministic, or whether it is probabilistic too and the distinction is one of dimensionality rather than kind.
Whether chaining probabilistic checks raises confidence or erodes it. Layered discrimination against compounding degradation, with both sides holding real numbers.
Whether an objective and key result is an incentive or a consequence, and whether quality can be expected rather than rewarded.
Whether describing these systems in human terms is a useful communication device or a category error. One position defended it as a way of making consequences discussable, while conceding it also makes replacement easier to sell. The other objected to it throughout, calling the technology a mimic of human memory with no judgment and no reasoning.