Your AI has a you-shaped hole in it
I think I've learned more about AI in the last two months than I did in the previous two years.
The biggest lesson is that deploying agents across multiple clients is hard. Getting an agent to work once is not the hard part. Getting it to keep working after you step away is.
Two problems show up after the demo works.
The first is visibility. Did the agent run? What did it do? Where did it get stuck? Why? The software industry calls this observability: using logs, metrics, and traces to understand what happened inside a system. Knowing whether the final work was good is a related job: evaluation and quality control.
The second problem is what happens when the tools surface the work. Somebody still has to look at it. Check every output yourself and one morning you have 50 completed tasks waiting for approval.
Who is working for whom here?
You are the missing infrastructure
When I work beside an agent, like the Chief of Staff I wrote about last week, it performs. When I let it run alone or hand it to someone who isn't me, it cracks. Sitting beside it, I do five jobs without noticing:
Monitoring. I notice when it is wrong. Nobody has to tell me.
Quality control. I fix errors before they spread.
Escalation. The weird cases come to me by default because I am right there.
Context. I feed it what changed: the annoyed client, the deal that moved.
Repair. When it breaks, I am already at the keyboard.
All five jobs are invisible. All five feel free. All five are being done by my attention. Autonomy does not remove them. It leaves them unfilled.
You already know this cliff. It is the one in your business. Everything runs fine while you’re at the center of everything. Then you try to step back or go on vacation and everything breaks. We’ve all read eMyth Revisited. We know what the answer to everything depending on us is. It’s to replace your own time and attention with systems, one painful system at a time.
Your AI cannot easily outgrow you for the same reason. There’s no system set up to govern and guide it: systems drift, models change, integrations break, and usage costs grow quietly. The work you have to do as an owner doesn’t go away, it changes shape.
Who watches the agents?
Over the next few weeks, I am going to write about these five jobs and how I am trying to take each one off my plate. The first is monitoring.
I have scheduled tasks in Claude and Codex. One sends me a daily brief based on upcoming meetings and emails. Another reviews the code in one of my apps each night.
With one task, I notice when the email does not arrive. Will I notice when I have 15? What if some run weekly or monthly? Silence could mean there was nothing to report, the credentials expired, the laptop slept, or the result landed somewhere I forgot to check. Every once in a while I get an email or an alert from an agent I forgot about. It’s usually something I built a while ago and abandoned but it was quietly running in the background and I didn’t notice it.
Agent observability is becoming a real product category. Claude and Codex provide pages for reviewing scheduled work. Microsoft, Google, and Amazon provide monitoring and tracing inside their clouds. Langfuse is an open-source option you can host yourself.
But each tool sees the part of the world connected to it. Azure does not automatically see a Codex task. Codex does not know what an agent did inside another cloud. Langfuse needs the agent to send it the right trace data.
Even inside Azure, my first dashboard only showed that an agent failed. The deeper explanation—tool calls, exceptions, token use, inputs, and outputs—appeared after I connected tracing and Application Insights. Eventually, I built a thin dashboard to pull the pieces together.
That proved the point: observability is not one product you buy. It is an operating system you assemble around the work you care about.
Start with a receipt
Before you schedule another agent, make sure you have a way to tell if it’s working. It should tell you:
What I did.
What I skipped.
What needs your decision.
Send it somewhere you already look like email or slack. Then add one rule: if the receipt does not arrive, that is itself an alert.
It is not a complete observability system. It is the first piece of infrastructure that begins replacing you.
We naively tell people to “use AI,” or to become “AI native,” but these admonitions fail to take into account that an inexperienced team deploying agents into an immature product environment is going to end up with a mess of half working agent spaghetti. I’m still working through mine.
Next week: how I am approaching quality control without turning myself into a 50-item review queue.