Wrong output from basic LLM chain with llama3.2

[Asadrz11014]

Llama3.2 model not cooperating with the prompt.

So I was following this video (https://www.youtube.com/watch?v=a8Dhj3Zh9vQ) to automate my gmail labelling. I have copied the first half of their workflow almost exactly. The only changes i have made are the label names in the structured output parser, the LLM model used, and the system prompt in the basic LLM chain block.

Here is my system prompt:

You are an expert email analyser. Your task is to categorise an email based on its main purpose. Focus only on the relevant content of the email and disregard unrelated information such as advertisements, boilerplate text, or legal disclaimers. Use the following categories to classify the email:

Categories:

1. Receipts/Confirmation Mail:
   Emails confirming a purchase, order, subscription, or registration. These emails usually contain information such as:

   • Confirmation of an order or payment.
   • Order number, transaction ID, or a receipt attached.
   • Keywords like “confirmation,” “receipt,” “order placed,” or “payment successful.”

2. Application Updates:
   Emails providing updates about an ongoing job application. Examples include:

   • Interview invitations or scheduling details.
   • Notifications about application progress, acceptance, or rejection.
   • Common phrases: “Your application status,” “Interview scheduled,” “Regret to inform,” or “Application under review.”

3. New Job Alerts:
   Emails suggesting new job opportunities or recommending job openings. These often include:

   • Job listings matching your profile.
   • Phrases like “job opportunity,” “new jobs for you,” “positions available,” or “apply now.”
   • Links or buttons to apply for suggested jobs.

4. Other:
   Emails that do not fit into the above categories. These could include newsletters, promotions, surveys, or unrelated content.

---

Examples:

Example 1:

Email Content:
“Thank you for your purchase! Your order number is #12345. You can track your shipment using the link below.”
Categorisation: Receipts/Confirmation Mail

Example 2:

Email Content:
“Your application for the Software Engineer position is under review. We will get back to you shortly.”
Categorisation: Application Updates

Example 3:

Email Content:
“We found 10 new job openings matching your profile. Apply now to kickstart your career!”
Categorisation: New Job Alerts

Example 4:

Email Content:
“Don’t miss our end-of-season sale with up to 50% off on select items!”
Categorisation: Other

---

Instructions:

1. Carefully analyse the provided email content.
2. Ignore irrelevant sections such as headers, footers, advertisements, or legal disclaimers.
3. Categorise the email into one of the four categories above.
4. Return only the category name as your response in JSON format:
   • “Receipts/Confirmation Mail”
   • “Application Updates”
   • “New Job Alerts”
   • “Other”

It seems as if the model disregards the prompt entirely and just outputs a random JSON label which is not listed in the prompt at all. For example: An email containing new job suggestions with a bunch of job links ended up having the LLM output “Complaints and Resident Engagement Officer” as the JSON label result.

Here is the workflow image:

This is my first post so apologies if the formatting is incorrect or if i have explained this problem poorly.

Information on your n8n setup

• n8n version: 1.72.1
• Database (default: SQLite):
• n8n EXECUTIONS_PROCESS setting (default: own, main):
• Running n8n via (Docker, npm, n8n cloud, desktop app): Docker with the self-hosting ai starter kit
• Operating system: Mac Apple silicon

[VA70]

Are there any follow up on this perhaps?

[Asadrz11014]

Hi, yes I’ve actually managed to finish the workflow and it works.
Here is the new prompt:

{{ $json.text }}{{ $json.headers.subject }}

You are an expert email analyser. Incoming emails need to be labelled. The available labels to associate to the email are:

1. Job_Alerts/Newsletters - The email contains new job suggestions or notifications from a job board.

2. Application_Updates - The email is giving an update on an ongoing application. Could be a rejection or information on progress in the application.

3. Receipts/Confirmation - Confirming a purchase, registration to a service or a 2-factor authentication email.

4. Miscellaneous - The email contents do not align with any of the other labels mentioned.

Your task is to analyse the email subject and contents and then choose an appropriate label to give it. The label must be one of the 4 mentioned above. Do not make up any labels.

Return the label name as a JSON object, nothing else. Do not return an empty object.

Ensure your output is a valid JSON object, not a string containing a JSON object. For example, output: { “labels”: “Job_Alerts/Newsletters” }. Do not surround the JSON object with quotes or additional formatting. Do not include any additional properties or formatting.

And this is what the workflow looks like:

I’ve found that sometimes it can still give an incorrect empty JSON object when the incoming email is too complicated for the llama model to understand, however, i figured that with the model I’m using, it is good enough.

I’ve just noticed the last node is not connected. That was simply for testing purposes. I’ve ran it with that node connected and it works fine.