Here's the situation I was trying to fix.
I had a task I did every week: a set of about a dozen checks before a report went out. Data pulled correctly, totals tie, the right filters applied, the summary matches the numbers, the recipient list is current. Nothing hard. But skipping one check produced a bad report, and a bad report cost me a conversation I didn't want to have.
So I built an AI agent to do it. The agent could read the data, run the checks, flag anything wrong, and even fix some issues. It worked—most of the time. Then it broke in a way I didn't catch, and the failure was worse than the one I was trying to prevent.
This piece is the honest postmortem. It's a Workflow Field Test, and the conclusion is the one I didn't want: for this task, a manual checklist was better.
Why the Agent Seemed Like the Obvious Answer
Three reasons, and they were all reasonable.
The Task Was Repetitive
A dozen checks, every week, mostly the same. That's the textbook case for automation. If a human does it, they'll get bored and skip steps. The agent won't.
The Checks Were Rule-Based
Not judgment calls—verifiable conditions. Totals either tie or they don't. Filters are either applied or not. This is the kind of logic that should be easy to delegate.
The Failure Cost Was Real
When a bad report went out, the cost was a distracted director, a re-send, and a small loss of trust. The agent was supposed to eliminate that failure mode by catching what a tired human missed.
All three of those were true. And the agent still lost to the checklist.
What Broke
Four failures, in order of how much they mattered.
Failure 1: The Agent's Errors Were Silent
The agent ran. It reported "all checks passed." And it was wrong—one of the checks had been evaluated against the wrong column because the source data had shifted by one row.
The checklist would have caught this, because the human step is "open the file and look at the totals." The agent had no equivalent moment of contact with reality. It evaluated a rule against data it had misread, and both the evaluation and the report said "fine."
This is the core problem: an agent can fail without failing visibly. A checklist that's skipped produces a known gap. An agent that's wrong produces a confident "passed."
Failure 2: The Exceptions Were Where the Work Was
The agent handled the twelve routine checks well. The exceptions—the one week when the data source changed format, the week when a new cost center appeared, the week when a filter was legitimately supposed to be off—were where the actual judgment lived.
The checklist forced me to be present for those, because I was running the checks. The agent ran the routine ones and reported an exception, but by then the cost of investigating the exception was higher than the cost of just having done the checks myself.
The agent automated the easy 80% and left the hard 20% harder, not easier.
Failure 3: The Maintenance Outgrew the Savings
The agent needed upkeep. When the source data changed, the rules needed updating. When a new check was added, it needed to be coded in. When the output format changed, the agent needed to learn it.
Each of those was a small task, and together they exceeded the time the agent saved. I was maintaining an agent to avoid doing a task that took fifteen minutes. The math never worked; I just didn't do it honestly until the third month.
Failure 4: The Trust Was Misplaced
This is the one that ended it. After the silent failure, I couldn't trust the agent's "passed" anymore. I started checking its work—running the checklist anyway. Which meant I was doing both.
The agent hadn't removed the task. It had added a layer that I had to verify, and a layer that had already proven it could lie.
Why the Manual Checklist Won
Once I dropped the agent and went back to the checklist, three things became clear.
The Checklist Has Built-In Contact With Reality
Every check requires opening something and looking at it. That contact is where errors get caught—not by the rule, but by the human noticing something the rule wasn't designed to see.
An agent evaluates rules. A checklist forces observation. For this task, observation was the point.
The Checklist Is Visible
When I run the checklist, I know I ran it. I can point to the moment. When the agent reports "passed," I have no equivalent—I have to take its word, and its word has been wrong.
For a task where the failure cost is trust, visibility matters more than speed.
The Checklist Can Change Instantly
When the data source changes format, I adjust the checklist in the moment. The agent needs updating, testing, and redeploying. For a task with a changing environment, the flexible tool beats the rigid one.
The checklist's weakness—that it depends on a human being present—is also its strength: the human is what adapts.
The Decision Rule
I don't think agents are bad. I think they're the wrong tool for this task, and I now have a rule for telling the difference.
Use an agent when:
The task is high-volume. Hundreds of repetitions, where the human cost of running the checklist is real.
The environment is stable. The inputs, formats, and rules don't change often.
The failure is detectable. When the agent is wrong, you'll find out quickly and cheaply.
The judgment is genuinely minimal. Not "mostly rule-based" but "actually rule-based," with exceptions that are rare and well-understood.
Use a manual checklist when:
The task is low-volume. A dozen checks, once a week.
The environment shifts. Formats change, exceptions are common, context matters.
The failure is silent and costly. When being wrong doesn't announce itself, you need a human in contact with the work.
The point is the observation. Some tasks are valuable precisely because doing them forces you to look.
The task in this piece hits all four checklist criteria. That's not a knock on agents. It's a recognition that "repetitive and rule-based" isn't the whole test.
What I Actually Kept
The honest accounting: I kept the checklist and dropped the agent. But not all of it.
The checklist, in a shorter form. Twelve items became eight, because some of them turned out to be redundant.
One agent-assisted step. The data pull itself is automated—that part was stable, high-volume, and detectable if wrong. That's the part the agent earned.
A written record of what each check is for. Not the check itself, but the reason. This made the checklist easier to prune and easier to hand off.
A monthly review of the checklist. Formats change, so the list needs to change with them.
The agent handled the plumbing. The checklist handled the judgment. The split was the thing I should have designed from the start.
What Still Needs You
The parts of this decision that can't be automated:
Honest measurement. I wanted the agent to work. It took three months to admit the math didn't. Measuring the maintenance, not just the task time, is a human discipline.
Recognizing the silent-failure category. Whether a task's failures announce themselves is a judgment about the task. The model can't know what "silent" means for your context.
Deciding what to keep. The agent did one thing well—the data pull. Splitting the task into "automate this part, checklist that part" is a judgment, and it's the one that would have saved me three months.
Writing the checklist at the right length. Eight checks, not twelve. Too many and you skip them; too few and you miss something. That's a judgment about your own failure modes.
The model can help build and maintain the automation. You decide where the automation ends and the judgment begins.
The One-Task Version
If you have a task you're thinking about automating:
Write the checklist first. Before any automation, list the checks in plain language. This is the baseline.
Categorize each check. Which are rule-based and stable? Which require observation or judgment?
Automate only the first category. Leave the rest manual. If nothing falls in the first category, don't automate.
Run it for two weeks. Track the maintenance time, not just the task time.
Compare honestly. If maintenance exceeds the savings, go back to the checklist. That's not failure—that's the test working.
The reflex is to automate the whole task. The discipline is to automate the part that earns it and keep the part that doesn't.
For this task, the manual checklist earned its place. The agent didn't.
Better work first. More options next.
Make the workflow earn its place.
No feedback yet — submit the first.