Human-in-the-Loop AI Development

I ended the previous article in this series with a question: does any agent in your setup still know things it shouldn't? This one is about a related but different failure — not what an agent knows, but when you actually stop it and check its work.

Automation doesn't have to mean losing control. But the way most people set up approval gates ends up giving them a sense of control without that control actually being effective.

The gate I stopped paying attention to

After wiring up approval steps between every stage of a pipeline, I noticed something uncomfortable: I was clicking "looks good" faster than I could possibly have read the diff.

There were gates everywhere — after the spec, after implementation, after verification, sometimes after a single file change. Reviewing all of them properly would have been a full-time job on its own, so without deciding to, I started skimming. Then not even skimming. Just approving, because the pipeline was waiting and I had other things to do.

Nothing about that setup was unsafe on paper. A human was involved at every step. But a gate nobody actually reviews isn't oversight — it's a formality disguised as a review. In a way, it's worse than having no gate at all, because it creates a false sense of security: I thought I had control over what reached production, but I didn't.

When to let the agent move on its own

Not every step deserves a stop-and-wait gate. Autonomy earns its place when a mistake is cheap, reversible, and easy to catch after the fact: running a test suite, applying a formatter, a refactor that already follows an approved spec and pattern, anything a CI check would catch anyway if it slipped through.

Gating these steps doesn't add safety. It adds friction that trains you to stop paying attention — which is exactly what happened to me.

When to insist on stopping it

The steps worth a real gate are the ones where being wrong is expensive, hard to reverse, or hard to detect later. Approving a spec before implementation locks in assumptions. Anything that touches production data. Anything you're about to present as finished to someone else. Architecture decisions that later work will build on top of.

The rule I use is simple: multiply the cost of being wrong by how late you would discover the error. If either factor is high, it deserves a real gate. If both are low, the gate is completely useless.

What "actually reviewing" looks like

The test I use now, before approving anything: can I explain, in one sentence, why this specific output is correct — not "it looks fine," but the actual reason it satisfies what was asked.

If I can't produce that sentence, I'm not reviewing; I'm simply giving it my approval. That's usually a sign the gate is in the wrong place, not that I need to try harder — either the step doesn't deserve a gate at all, or it bundles too many things together to review them properly in one pass.

A few gates taken seriously are better than many ignored

The instinct, once something goes wrong, is to add another approval step. That's usually the wrong fix. More gates spread your attention thinner, and that lack of attention is exactly what makes a gate dispensable.

The better fix is fewer, well-placed gates — at the points where being wrong actually costs something — with the discipline to give each one real attention instead of clicking without thinking. Batch the trivial approvals or skip them entirely. Save deliberate review for the moments where it changes the outcome.

Conclusion

The goal was never for the agent to decide instead of me. The goal is for it to work inside a system where I still control the decisions that actually matter — and where the gates I've built are honest about which decisions those are.

A pipeline with a human involved at every step isn't necessarily safer than one with fewer, better-defined gates. It's only safer if that person is actually paying attention.

In the next article of this series I'll explore what happens when that discipline slips: Code Slop and AI-Generated Technical Debt — how to recognize output that passed every gate and still wasn't actually good.

Until then, the question worth asking about your own setup isn't:

Do I have a human in the loop?

It's:

The last time I approved something, could I have explained why it was right?

If you've found a gate placement that actually works for you, I'd like to hear where you put it — feel free to reach out.