Workflow OpenAi Assistant -> Answer Questioms Off Photos -> Feedback Loop To Train Assistant(Fine Tuning or Knowledge Base?)

As the title says. Im trying to find a way to feed N8N a JSON file thatll have photos, questions with potential answers, and to answer those. I have a system message for what the Assistant is expected to do, and each question will have a separate prompt to queue up Ai go review photos to answer that question. At times its wrong so an employee will fix it which i want to send that feedback to learn. Any ideas how we could do that. We can do webhooks of course to streamline that.

Hey @BrettDev, welcome to the community! :wave:

Here’s a possible approach to structure your workflow in n8n

Not sure what exactly you want to do but the rough structure looks like this:-

  1. Receive and Process Input JSON

    • Use a Webhook or HTTP Request trigger to ingest a JSON containing:

      • Photos (URLs or base64)
      • Questions
      • Potential answers
      • Metadata (e.g., user ID, timestamp)
  2. Generate AI Responses

    • For each question, construct a prompt incorporating the system message and relevant photo.
    • Utilize the OpenAI node to generate answers.
  3. Present AI Response for Human Review

    • Use the Human in the Loop node to send the AI-generated answer to a human reviewer via a platform like Slack, Telegram, or Email.

    • Configure the node to allow the reviewer to:

      • Approve the answer
      • Reject the answer
      • Edit the answer
  4. Capture Human Feedback

    • Store the original AI response, the human-modified answer, and the reviewer’s decision (approve/reject/edit) in a database or data store.
  5. Update Knowledge Base (May be)

    • Based on the feedback:

      • Add the new information to your knowledge base for future reference.
  6. Provide Feedback to Users

    • Inform the user about the status of their query, whether it’s been approved, rejected, or edited.

For a practical example, n8n provides a template for a simple email response system that incorporates human feedback:

:point_right: A Very Simple “Human-in-the-Loop” Email Response System Using AI and IMAP

That’s all I have, hope this helps. : )