AI Lies. Engineers Prove

An AI agent can tell you the tests passed without running them. It can say a bug is fixed when all it did was edit a file. It can call a release live because the push succeeded.

The report sounds finished. The work might not be.

I call that lying in the title because that’s how a false completion report lands when you’re relying on it. I’m not making a claim about intent. For the engineering decision, the important part is whether the claim holds up.

Show me what ran

If an agent says it tested a change, I need to know which command it ran and what came back. Not the command it recommends I run. Not sample output showing what success would look like.

Actual execution output, including failures.

That still takes interpretation. A successful command might have skipped the relevant tests. It might have run against an old build or the wrong environment. The evidence needs to match the claim, not just look reassuring.

A type check passing means the type check passed. It doesn’t tell me the checkout flow works. A test covering one input doesn’t establish every case. Engineers don’t get to stretch that evidence either.

Read back the thing that changed

Suppose an agent is asked to update a setting through an API. The request returns successfully and the agent reports it’s done.

I’d still read that setting back from the target system. Did the right record change? Is the value what we asked for? Was the request accepted for processing, or has the change actually happened?

Publishing has the same problem. A successful push says the repository received the commit. It doesn’t say the deployment finished or that the public page contains the new content.

Open the actual URL. Check the body, not just the status code. A page returning 200 can still be the old page.

Asking again isn’t verification

“Are you sure?” isn’t much of a check. The same agent can give you another confident answer based on the same missing evidence.

A second agent can help review the reasoning. It can spot a gap or suggest a better test. But if both agents are reading the first agent’s summary, they’re still working from one unverified report.

Give the reviewer the code, the requirements, and the actual results. Let it inspect what happened. Agreement alone doesn’t make the claim true.

And when a check can’t run, that needs to stay visible. Missing access, a failed install, a blocked command. None of those should turn into plausible-looking output so the task can be called complete.

Say what you know

“Implemented, not tested” is useful. So is “local checks passed; production hasn’t been verified.” I can make a decision with that information.

“Done” hides too much when it stands in for all of those steps.

The word “prove” in the title isn’t a promise of perfect software. It’s a demand to back up the claim with a check that someone else can inspect, and to be honest about where that check stops.

I use AI because it helps me get work done. I still want the command result, the changed record, or the live page before I accept its completion report.