Ask a model whether it should be writing your production code

Here's an experiment that takes about two minutes.

Ask a frontier model, plainly and without leading it, whether it should be writing production code with minimal human oversight in a system where failure carries real consequence. What comes back is a fairly consistent no. Some cases are fine: prototyping, scaffolding, bounded work a person will actually read. Production systems where a mistake costs something, considerably less so.

Your first instinct should be to distrust that result, and you'd be right to. Models are extremely sensitive to framing. Ask a question in a context where caution is the salient theme and caution is what you'll get back. This is not a system expressing a considered judgement about its own capability. It is a system producing the output it was trained to produce for that class of question. As evidence about what AI can actually do safely, it's worth very little.

But that's what makes it worth looking at. What surfaces in that answer isn't the model's view. It's the safety posture of the organisation that trained it. The labs building these tools have deliberately taught them to decline to endorse their own unsupervised use in consequential settings, and that decision tells you something.

It sits awkwardly next to how the same tools are marketed, priced, and rolled out. The product positioning points toward substantial autonomous capability. The trained behaviour points toward "get a human to check this properly." Both come from the same organisation. Whatever these companies know about the limits of their systems is showing up in the guardrails more candidly than in the go-to-market.

There's an escape hatch people reach for here, and I don't think it survives contact. When people hear "high-stakes" they think aviation, medicine, nuclear, and conclude it doesn't describe them. But a production system that generates revenue is high-stakes almost by definition. It moves money, holds customer data, or underpins something a business depends on continuing to work. The bar isn't whether anyone dies. It's whether failure carries consequence you'd struggle to explain afterwards. By that measure most production software qualifies, and most organisations running it have quietly decided otherwise without ever examining the assumption.

What I actually think

I don't think AI shouldn't write code. It plainly can, and often usefully. What I think is that it shouldn't be leading.

Architecture, orchestration, the judgement about what a system needs to do and how its parts fit together: that is human work, and there are good reasons to keep it human. Generating code for a well-scoped task that a person has already framed is a different activity from letting the model set the shape of the solution and reviewing whatever comes back. Most organisations aren't distinguishing between those two things, and the second has quietly become normal while everyone continues describing the first.

That drift is what I want to look at, because it's where the assurance problem I wrote about last time stops being abstract. Nobody sat in a room and agreed to hand over the design of their systems. It happened one convenient task at a time.

Why the safety-critical comparison only half applies

I've leaned on aviation, nuclear, and medical devices as the benchmark for what real assurance looks like. It's worth being honest about where that comparison stops being useful.

Those industries developed their rigour with a structural brake in place. Sometimes the brake was an accident that made the cost of failure undeniable. More often it was simply physical: building an aircraft or a reactor takes years, and those years gave regulation and practice time to catch up before the next generation shipped.

Software has never had that brake. It was already fast. Continuous deployment, rapid iteration, ship-then-patch, all of it predates AI by a decade or more. What matters is that none of the old constraints have been removed. AI hasn't replaced a slow process with a careful one. It has added throughput to an environment that was already outrunning its own governance. The acceleration compounds rather than substitutes.

There's a less comfortable thing underneath this, and I'll state it as my own view rather than pretend it's settled. A very large share of current economic expectation is riding on AI delivering. Not just the valuations of the companies building it, but the growth assumptions of everyone planning to benefit downstream. If that bet fails badly, the correction isn't confined to a sector. That creates adoption pressure with no relationship whatsoever to whether any particular deployment is sound. It exists far above the level of any engineering team and presses down on decisions made several layers below it.

In fairness to almost everyone involved

I want to be careful here, because it would be easy to write this as though organisations are being reckless and their leaders are fools. I don't think that's what's happening. Having worked inside organisations navigating this, what I've seen looks much more like people making defensible decisions under conditions that make the defensible decision the wrong one.

Put yourself in the position of someone running a technology function right now. Your competitors are publicly claiming significant productivity gains. Your board has read the same articles you have. If you adopt aggressively and it goes badly, that's a problem in eighteen months. If you don't adopt and your competitors pull ahead, that's a problem at the next quarterly review. The incentives aren't subtle, and they aren't irrational. They're just badly calibrated to a risk that hasn't materialised visibly yet.

The engineers are in a harder position still, and this is the part that gets discussed least. They're being asked to change how they work, substantially, and then asked again a few months later because the tooling has moved. Anyone who developed a decent working method with one generation of models has watched it become half-relevant with the next. The context windows changed. The agentic capabilities changed. The right way to structure a task changed. Guidance written in good faith six months ago is now quietly misleading, and nobody has the time to rewrite it.

That matters more than it sounds. Professional judgement is built by doing something repeatedly and learning where it fails. It's very hard to develop judgement about a thing that keeps becoming a different thing. We are asking people to be responsible for outcomes produced by a tool whose behaviour they've had no stable period in which to understand. Then, in a number of organisations, we're measuring them on how enthusiastically they use it.

None of that excuses the outcome. But an argument that treats this as carelessness will be wrong about the cause and therefore wrong about the fix.

Absences stacked on absences

Ask why an organisation can't give a straight answer to "is this use of AI sound," and you won't find one clean failure. You'll find several gaps sitting on top of each other.

Most organisations have no internal expertise in this. Not security in the traditional sense, which is usually well covered, but the specific behavioural and assurance questions that arise from non-deterministic systems. That expertise exists, but it is concentrated in a small number of research labs and safety-focused organisations. It is not distributed across the far larger population of companies actually deploying the technology. The people who understand the problem best are almost entirely not in the rooms where the decisions get made.

There's no external standard to fall back on either. Regulation and industry practice for AI-generated code specifically, as distinct from AI systems in general, barely exists. An organisation that genuinely wanted to do this properly would struggle to find much established guidance to point at.

And underneath both of those is something worth saying plainly, because it tends to get lost in product conversations: the underlying safety and alignment questions are not solved. Not by the labs, not by academia, not by anybody. Making these systems reliably do what was intended, and making it possible to verify that they have, remains open research. We are deploying at enormous scale, into consequential settings, a technology whose foundational assurance properties are still being worked out by the people who built it. That isn't a criticism of those people. It is simply the position we're in, and organisational decisions are being made as though it weren't.

Together these mean organisations aren't making an informed decision about where AI belongs in how they build software and where it doesn't. They're proceeding because nothing in the system is positioned to draw that line for them, or to tell them clearly what sits on either side of it.

What the drift looks like from inside

Individual teams end up working this out alone, because there's rarely a function whose job it is to think it through. The result is a landscape of quiet improvisation: some teams careful, some enthusiastic, no consistent standard between them, and no shared understanding of which tasks warrant which level of scrutiny.

A centralised function could address this. Something structured like a platform or developer-experience team, but oriented specifically around AI: building shared guidance, understanding the risk profile of different categories of use, maintaining that guidance as the tooling shifts underneath it, setting a bar so that every team doesn't have to invent one privately. Very few organisations have built this, because it reads as overhead and doesn't show up in any adoption number.

Which points at a related problem. The metrics being tracked are almost entirely about adoption itself. How many engineers are using the tool. What proportion of changes involve generated code. None of that says anything about whether the usage was appropriate to the task, adequately reviewed, or safe. An organisation can hit every adoption target it sets and be no closer to knowing whether any of it was a good idea. Measuring the wrong thing confidently is worse than measuring nothing, because it manufactures the appearance of diligence. It also quietly instructs everyone about what the organisation actually values, which is usually not what the accompanying memo said.

There's a practical constraint that rarely gets said out loud, too. Context limits are real, and they shape behaviour in ways that reduce rigour. When usable context is finite, the pull is toward compression: summarise rather than read in full, trust a condensed representation of a larger change, accept the model's account of what it did instead of checking. That's a sensible adaptation to a genuine limitation. It's also precisely the shortcut that assurance exists to prevent.

Does regulation change the picture?

It should, and it partly does. Regulated environments come with machinery most software organisations lack: change control, independent validation, audit trails, people whose actual job is to ask uncomfortable questions before something ships. If AI-assisted development is going to be scrutinised properly anywhere, it ought to be there first.

I'm less confident it works out that way in practice. The pattern I've watched repeatedly is that regulated industries produce genuine rigour in the areas where regulators are explicitly looking, and something much closer to compliance theatre everywhere else. Cybersecurity is the clearest recent example: mature frameworks, widely available, and an industry that has largely implemented the certification rather than the practice.

AI-assisted development currently sits in the "nobody is explicitly looking" category, even in heavily regulated firms. So regulated status offers the machinery but not necessarily the attention. Whether that machinery gets pointed at this problem before or after something goes visibly wrong is an open question rather than a safe assumption.

The consequence I think gets least attention

If engineers increasingly work with code they didn't write and don't fully read, the organisational capacity to notice when the model has done something wrong erodes. Slowly, and without any obvious moment where it happened.

That's a second-order problem sitting on top of the first one, and it's the more serious of the two. The immediate risk is that AI produces something incorrect. The compounding risk is that the population of people capable of recognising it gets smaller every year.

It connects directly to what assurance actually requires. Independent evaluation, reproducible behaviour, defined criteria: every one of those depends on somebody being able to look at the system and form a competent judgement about it. Erode that capability across an organisation and you haven't weakened one control. You've removed the substrate the others rest on.

I don't think this shows up in any incident report. It shows up five years later, in the calibre of judgement available when it's finally needed.

Where I've landed

Nobody is going to stop using AI to write code, and they shouldn't. But there's a position between "AI-first" and "never" that most organisations currently occupy neither side of.

AI-enhanced, not AI-led. Human judgement sets the architecture and decides how the pieces are orchestrated. AI gets reached for on a task-dependent basis, to produce code for well-scoped work that a person has already framed and can meaningfully evaluate. That's a collaborative model, not a defensive one. What it rules out is the default posture where AI is the first move regardless of what's being asked, which is the direction most tooling and nearly every incentive is currently pushing.

Held up on its own, that degree of deliberateness can sound faintly paranoid. But engineering organisations are already fastidious about far less consequential things. Access control. Change management. Deployment gates. Nobody calls those paranoid. Asking why AI should be the one place we relax the instinct seems a reasonable question rather than an anxious one.

Alongside that, deliberate governance at the organisational level rather than every team improvising privately. Not heavyweight, but present, and owned by someone whose actual job it is. Above that sits the question of real regulation and industry-wide standards, which is genuinely political both inside companies and eventually at national level. Whether the limiting factor there is research or simply political will is a question I want to come back to properly, because I don't think the answer is obvious and it deserves more than a paragraph.

The thing I keep returning to is that none of this was decided. No organisation I'm aware of held a meeting and resolved that AI should set the shape of their systems. It arrived one convenient task at a time, under real pressure, handled by people doing their best with a tool that kept changing. The absence of a decision has been mistaken for the absence of a question.

The models, for what little it's worth, would tell you the question is still there. So would most of the engineers, if anyone asked them.

Seiya Alger-Hilton is an engineering manager with twenty years in regulated financial systems. Hilton Labs is his outlet for thinking carefully about AI assurance and governance. Correspondence welcome: seiya@hiltonlabs.org