Topics arrive by webhook or on a weekly schedule. Per topic: research through a search API, outline, draft, then a QA gate written in code, not in a prompt: minimum length, required sections, a sources section with two or more links, and a list of banned marketing claims. A failing draft goes back with the reasons and is never published. A passing draft goes to a human on Slack and, on approval, is published as a draft to Google Docs or the CMS. Token usage and cost are accounted per run.
The gate is the point. Without it the approver reads everything; with it they read the ones that cleared the bar.
Test run on sample topics: three researched, drafted, checked and published as drafts in one run; a fourth failed the gate and came back with the reasons.
The part I took from your post is that the gate is in code and not in the prompt. I built the opposite first a badge that flagged any run which processed nothing and a reviewer killed it inside an hour, because a quiet day with zero processed is simply correct. Your banned-claims list is the bit I cannot work out: has the gate ever failed a draft that was actually fine, and how did you find out it had?
Does the gate check that each claim is supported by its source, or just that the links are present? I’d include the source excerpts in the approval bundle, alongside the exact draft version. That gives the approver something concrete to check beyond length and required sections.
Aghassi, on the banned claims list: it is a plain lowercased substring match over four phrases, guaranteed, risk-free, 100%, best in the world. Three of those are safe. 100% is not, because it turns up in sentences that are simply factual, a 100% owned subsidiary, or 100% of these companies still file a return. So the false positive is designed in rather than hypothetical.
What makes it tolerable is that a failing draft comes back with the reason attached, so the reviewer sees “banned claim: 100%” sitting next to a sentence that is obviously fine and overrides it in a second. The gate never publishes, it only returns, so a wrong fail costs one round trip and nothing else.
Your badge for runs that processed nothing is the sharper lesson in the other direction. A quiet day with zero processed is a correct result, and alerting on it trains people to ignore the alert.
smallfixeslab, to your question: just that the links are present, and cruder than that. It counts occurrences of http in the body and wants two, and separately requires a Sources heading to exist. Two links anywhere in the text with an empty Sources section passes. Nothing maps a claim to its source.
Your fix is the right one and it is cheap, because the research step already holds the excerpts. Putting them into the approval bundle alongside the exact draft version gives the approver something concrete to check against instead of a word count. Matching claim to source in code is the harder half, and I would rather not have code guessing at that.
That is a better answer than I expected. The reason attached to the fail is what makes it cheap. I hit the same substring problem yesterday, the other way round. A page scorer I use matched short abbreviations without case, so the word “do” counted as a credential and pages passed that should have failed. It showed no reason, the score just went up, so nobody could catch it in a second. A wrong fail with a reason costs you one round trip, and a wrong pass with no reason costs nothing you can see.