Verification pipeline case study: catching AI content errors

Problem

I built an ecommerce content site to test how far an AI content operation could actually run on its own. Before writing a single pipeline stage, I checked something basic: were the product prices right?

Twelve prices, checked against source. Twelve wrong. Not one was correct.

None of them looked wrong. That is the whole problem with generated content in a commercial context. A price that is confidently stated and quietly incorrect does not read as a mistake, it reads as an article, and it stays wrong until a buyer discovers it the expensive way. Any content system I built on top of that foundation would have been a machine for publishing errors faster.

So the content pipeline waited, and I built the thing that has to exist underneath it.

What I built

An autonomous verification pipeline that runs overnight on a headless server and treats every factual claim as unproven until it is fetched from source.

The loop: pull the next unit from the worklist, fetch the source, record the claim alongside the source URL it came from, run the gates, commit that unit on its own, and produce a morning report. Anything the machine cannot fetch, it does not guess at. It escalates into a queue I clear by hand in a real browser, which is the honest answer for sources that block automation rather than a silent fallback to whatever the model remembers.

Seven overnight sessions are logged so far in a ledger that now runs to several hundred lines.

How it works

Three design rules do the work:

The autonomy line is drawn at cost, not at difficulty. Fetching, counting, gating, and reverting are fully automated, because a machine doing those badly is cheap to catch and cheap to undo. Publishing, merging, and any claim that could put a wrong recommendation in front of a buyer are not automated, and are not meant to be. Six standing human decision points are written into the process by name. The system knows what it is not allowed to decide.

Nothing merges into the live site by accident. Work lands on an unpushed branch, commits one unit at a time, and passes five gates before it goes anywhere. A failing unit reverts itself rather than degrading quietly. The mobile rendering gate runs in a real browser.

Every claim carries its source. The claim and the URL it was fetched from are recorded together, so any fact on the site can be traced back to where it came from, and a stale source is a thing I can re-check rather than a thing I have to remember.

The buildmysimrig home page. The headline reads every part checked against the part it bolts to, above three platform entry buttons for PC, Xbox and PlayStation. Below, a row of build cards for under 500, 800 and 1500 dollars, each showing a total price and its wheel base, wheel, pedals and mount, and each carrying an unverified badge. — Figure 1. The verification state, put on the page instead of kept in the ledger. Each build lists every part against the part it bolts to, and each one still carries an unverified badge, because the mounting patterns behind these three have not been checked against the manufacturer yet.

Results and impact

The finding above is the result that matters. Twelve of twelve prices wrong is not a fun fact, it is the argument for the entire piece of infrastructure, and it is the number I would put in front of anyone who thinks AI content is a volume problem rather than a verification problem.

A published buildmysimrig guide, best sim racing setup under 800 dollars for PS5. A picks panel lists three products, each with its price, an as-of date of 29 July 2026, and a green FIT stamp. A note under the panel explains that the stamp covers the product's own interface, platform and price claims rather than compatibility between parts. The section below, how these were selected, states that twelve catalogue prices have been checked and twelve were wrong. — Figure 2. The same discipline reaching the reader. Every price is stamped with the date it was checked, the FIT mark says exactly what it does and does not cover, and the piece states the error rate that produced the rule instead of asking to be trusted.

Alongside it, the pipeline instruments its own cost: a human-minutes log currently at 50 minutes across nine units, with the per-unit trend falling as the automated stages take more of the work. That figure is the point. It tells me whether the system is actually earning its keep, and it is the number that will decide whether the full content pipeline gets built on top of it or scrapped.

The site is live with its first articles indexed and the content pipeline specified but deliberately unbuilt. That order is the decision I would defend hardest here: verification first, volume second. Build the part that catches you being wrong before you build the part that lets you be wrong at scale.