How to save LLM input tokens (don't repeat the whole prompt for each row)?

Questions
1

Couldn’t find an answer to this using Google and the internal search here.

Describe the problem/error/question

I am using the Basic LLM Chain node and iterate over Google Sheets with thousands of rows. The node uses the same long prompt for each row, while only one field of the Google Sheet is used as a variable in the prompt.

That feels like a waste of input tokens. Using the chat interface of an LLM I would begin the chat with the main task and for each row of the sheet I would only input the variable. The OpenAI API for example offers conversations for that purpose.

Is that achievable somehow?

Information on your n8n setup

  • n8n version: Running version [email protected]
  • Running n8n via (Docker, npm, n8n cloud, desktop app): Cloud
2

Try CAG - read the entire dataset before the agent and send it all at once to the AI Agent. This way data will be cached at the agent once and it will be processed in one shot.
Filter out un-necessary details to cut down on llm token
use a gpt model that allows for a lot of tokens ( Gemini ) etc

let me know how it goes and we can tweak it further.