Llama3.2 model not cooperating with the prompt.
So I was following this video (https://www.youtube.com/watch?v=a8Dhj3Zh9vQ) to automate my gmail labelling. I have copied the first half of their workflow almost exactly. The only changes i have made are the label names in the structured output parser, the LLM model used, and the system prompt in the basic LLM chain block.
Here is my system prompt:
You are an expert email analyser. Your task is to categorise an email based on its main purpose. Focus only on the relevant content of the email and disregard unrelated information such as advertisements, boilerplate text, or legal disclaimers. Use the following categories to classify the email:
Categories:
-
Receipts/Confirmation Mail:
Emails confirming a purchase, order, subscription, or registration. These emails usually contain information such as:- Confirmation of an order or payment.
- Order number, transaction ID, or a receipt attached.
- Keywords like “confirmation,” “receipt,” “order placed,” or “payment successful.”
-
Application Updates:
Emails providing updates about an ongoing job application. Examples include:- Interview invitations or scheduling details.
- Notifications about application progress, acceptance, or rejection.
- Common phrases: “Your application status,” “Interview scheduled,” “Regret to inform,” or “Application under review.”
-
New Job Alerts:
Emails suggesting new job opportunities or recommending job openings. These often include:- Job listings matching your profile.
- Phrases like “job opportunity,” “new jobs for you,” “positions available,” or “apply now.”
- Links or buttons to apply for suggested jobs.
-
Other:
Emails that do not fit into the above categories. These could include newsletters, promotions, surveys, or unrelated content.
Examples:
Example 1:
Email Content:
“Thank you for your purchase! Your order number is #12345. You can track your shipment using the link below.”
Categorisation: Receipts/Confirmation Mail
Example 2:
Email Content:
“Your application for the Software Engineer position is under review. We will get back to you shortly.”
Categorisation: Application Updates
Example 3:
Email Content:
“We found 10 new job openings matching your profile. Apply now to kickstart your career!”
Categorisation: New Job Alerts
Example 4:
Email Content:
“Don’t miss our end-of-season sale with up to 50% off on select items!”
Categorisation: Other
Instructions:
- Carefully analyse the provided email content.
- Ignore irrelevant sections such as headers, footers, advertisements, or legal disclaimers.
- Categorise the email into one of the four categories above.
- Return only the category name as your response in JSON format:
- “Receipts/Confirmation Mail”
- “Application Updates”
- “New Job Alerts”
- “Other”
It seems as if the model disregards the prompt entirely and just outputs a random JSON label which is not listed in the prompt at all. For example: An email containing new job suggestions with a bunch of job links ended up having the LLM output “Complaints and Resident Engagement Officer” as the JSON label result.
Here is the workflow image:
This is my first post so apologies if the formatting is incorrect or if i have explained this problem poorly.
Information on your n8n setup
- n8n version: 1.72.1
- Database (default: SQLite):
- n8n EXECUTIONS_PROCESS setting (default: own, main):
- Running n8n via (Docker, npm, n8n cloud, desktop app): Docker with the self-hosting ai starter kit
- Operating system: Mac Apple silicon