Don't tell the AI not to touch money. Make it so it can't.

1 August 2026 ·3 min read

• building with AI
• AI agents
• safety

I built an AI co-pilot for an app of mine. It was created to do real operations in the app: it reads what’s happening, drafts messages, and tees up actions for a person to approve. Early on, I had to answer one question:

Should it be allowed to move money?

The easy answer, and why I didn’t take it

The easy answer is: let it move money, and just tell it to be careful. Write a clear, firm instruction in the prompt. “Never charge a customer without approval. Never move funds you shouldn’t.” Models follow instructions. Mostly.

But that “mostly” is dangerous. The rule is clear and sensible. Surely the model will follow it?

An instruction to an AI is a request. It’s a wish you send into a system that reads it, interprets it, and usually does what you meant.

But a model can misread the instruction. It can be handed new text halfway through a task that quietly argues it out of the rule. For example, people can write “ignore your previous instructions” into content the AI reads, and sometimes it works (this is what’s called prompt injection). An agent “usually doing the right thing” is a fine bar for suggesting a headline. It is not the bar for moving someone’s money.

What I did instead

So I made the money-move impossible to do, not just “forbidden.”

Here’s how I did it. My co-pilot is allowed a fixed set of actions:

• read the state of a deal
• draft a message
• tee something up for a person

Each action is a tool it can pick up and use. Moving money would be one more tool like the others.

I set the code up so that tool cannot be attached to the co-pilot at all. If I ever tried to wire a money-move onto it, or an AI helping me write the code tried to, the code would refuse to build. It’s like a plug shaped so it won’t fit the socket: you can’t force it in, and you can’t forget and leave it in.

Expand figure to full screen

This is called type-driven design. A type is the shape a piece of code is allowed to take. You design the shapes so a dangerous state simply can’t exist. In this example, it’s so the agent can never move money. Break that rule and the code won’t compile, which means the app can’t even start. The mistake is caught before the program runs, not after it has done any damage.

Another name for this is poka-yoke, a term from the Toyota Production System (coined by Shigeo Shingo). Same idea: shape the connector so it physically can’t go in wrong. An AirPods case is a good example. The left and right sockets are shaped so the right earbud can’t fit in the left one, or the other way around.

Ask for behaviour or design the impossibility?

There are two ways to make software safe. You can ask for the safe behaviour: write the rule, trust the thing to follow it, and hope. Or you can design so the unsafe thing can’t be expressed at all: shape the code so the dangerous move fails to build, fails to save, fails to even be written down.

Expand figure to full screen

Asking scales with trust. Designing scales with nothing.

The real question

I don’t think the interesting question about an AI agent is “how smart is it.” How smart it is really comes down to what it can do. So the real question is: what is it allowed to do, and who guarantees that?

A prompt is not a guarantee. It’s the most persuadable part of an agentic system, and on the paths that touch money, or anything you can’t take back, persuadable is exactly what you can’t afford.

So on those paths I stopped writing rules for the AI to follow. I started building walls it can’t ask me to move.