How I stop an AI support bot from promising refunds: a policy guard pattern for n8n

Built with n8n
1

I built an AI customer support desk in n8n (37 nodes): it reads tickets from a support inbox or a form, classifies them with Gemini, looks up the order in Shopify, drafts a reply from the business’s own knowledge base, logs everything to a Data table, checks SLAs hourly and sends a daily report.

The part that took the most thought was not the AI. It was deciding when the AI is allowed to hit Send.

An LLM will happily write “your refund has been approved” to an angry customer. So between the AI draft and the Send Email node there is one Code node, the Policy Guard, that collects reasons to hand the ticket to a person. If the list is empty, the reply is safe. If not, it escalates with the draft ready.

const reasons = [];
if (!reply) reasons.push('no AI draft');
if (!s.autoSendIntents.includes(t.intent)) reasons.push(`"${t.intent}" tickets always go to a person`);
if (t.needs_human) reasons.push('classifier asked for a person');
if (reply && conf < s.minConfidenceToSend) reasons.push(`draft confidence ${conf} is below ${s.minConfidenceToSend}`);
if (t.sentiment === 'angry') reasons.push('customer is angry');
if (t.urgency === 'urgent') reasons.push('urgent ticket');
if (/\[[a-z ]+\]|\{\{|lorem/i.test(reply)) reasons.push('draft contains a placeholder');
if (/refund (has been|is) (approved|issued|processed)|discount code|free of charge|compensation|we guarantee/i.test(reply)) reasons.push('draft makes a commitment');
if (Array.isArray(d.promises) && d.promises.length) reasons.push('draft promises: ' + d.promises.join('; '));
if (t.contacted_last_7_days >= 2) reasons.push('repeat contact this week');
const decision = reasons.length ? 'escalate' : 'auto_send';

A few things I learned along the way:

  1. Allow-list intents, don’t block-list them. Only “order status”, “shipping” and “product question” can ever auto-send. Anything new or unexpected defaults to a human.
  2. Ask the model to list its own promises. The draft prompt returns a promises array next to the reply. If it is not empty, the ticket escalates. The model is surprisingly honest about this.
  3. Keep a regex net anyway. The model sometimes promises without admitting it, so phrases like “refund has been approved” or “discount code” are caught in code.
  4. Repeat contacts are a signal. Someone writing for the third time this week should get a person, even if the question looks easy.
  5. Ship it in draft mode. autoSendEnabled is off by default, so every “safe” reply goes to the team first. You switch it on once you trust the drafts.
  6. Tell the human why. The escalation email starts with “Why you are seeing this: urgent ticket; customer is angry”, plus the draft and a one-click “review and send” link. That alone saved more time than the auto-replies.

Tested end to end with real tickets: an order question answered with tracking, an angry damaged-item refund escalated as urgent, an out-of-office filtered out, a care question answered from the knowledge base, a Spanish question answered in Spanish.

Happy to answer questions about the guard or the rest of the flow. If anyone wants the full workflow, it’s on my page: Nasarhayatt | Whop

2

“We’ll put the money back on your card” walks straight past the commitment regex, so that check is still a block-list. The promises array is the stronger check, but it fails open: if the model returns a string or null, or drops the field, Array.isArray is false and nothing gets pushed. I’d flip it so anything other than an empty array is a reason to escalate.

3

@Ricart_Juncadella good catch on both, thank you.

You’re right that the regex was really a block-list of phrases, and that the promises check failed open. I’ve changed both:

if (/refund|reimburs|money back|back (on|to) your (card|account)|credit (it|your|the)|store credit|replacement|free of charge|compensat|discount|voucher|coupon|waive|guarantee/i.test(reply)) reasons.push('draft mentions money, refunds or compensation');
if (!Array.isArray(d.promises)) reasons.push('AI did not report its promises (fail closed)');
else if (d.promises.length) reasons.push('draft promises: ' + d.promises.join('; '));

The money check now matches the concept rather than specific sentences, and anything other than an array for promises escalates. Tested with your “money back on your card” example plus a null and a string promises: all three escalate, and a plain order-status reply with tracking still auto-sends.

It makes the guard more conservative (more tickets go to a person), but for anything touching money that’s the right trade-off. Fail closed is the right default here.