I’m running a WhatsApp bot on AI Agent (Tools Agent) + OpenAI Chat Model node (gpt-4.1-mini, Responses API enabled, File Search built-in tool for RAG) + Postgres Chat Memory + 3 custom tools (Call n8n Workflow Tool type).
My system prompt is fairly large (~8-9k tokens of mostly static instructions and reference examples), so I expected a healthy prompt caching rate since none of that content changes between requests. Checked the OpenAI usage dashboard and the cache hit rate is sitting at around 2%.
I pulled the raw input payload from the OpenAI Chat Model node’s execution data and found the cause: the entire system prompt, the dynamic conversation context block (customer name, session status, working hours, etc.), and the full Human/AI turn history are all flattened into a single "System: ..." string before being sent to the API, rather than being passed as a structured messages array with separate role-tagged objects (system / human / ai / tool).
Because the dynamic context block sits between the static instructions and the actual conversation turns, and that block changes on every single request, the prefix match breaks for everything that follows it — so caching never engages even though the bulk of the prompt (the static instructions) never changes.
From what I understand of LangChain’s AgentExecutor, the memory wrapper defaults to stringifying history unless return_messages=True is explicitly set, and the prompt should ideally use a ChatPromptTemplate with a MessagesPlaceholder to preserve history as proper message objects. I don’t see either of these exposed as configurable options anywhere in the AI Agent node’s UI.
Question: Is there any supported way (hidden expression, node setting, or different memory node configuration) to get the built-in AI Agent node to pass chat history as a structured array instead of flattening it into the system prompt string? Or is this a known limitation of the current Tools Agent implementation?
The clue to prove is the first token block that changes. Run one tiny A/B on the same WhatsApp session: one call with Postgres Chat Memory disabled, one with it enabled, then compare the raw input payload and cache_creation_input_tokens / cache_read_input_tokens.
If the disabled run keeps the 8-9k instructions as the stable prefix and the memory-enabled run injects customer/session/history text before that prefix, your cache miss is not an OpenAI cache setting problem. It is the message assembly layer. Post just that redacted diff and the two cache counters; no customer text needed.
I understand the issue now thanks for explaining it in detail.
What’s happening is basically not a model problem, but a prompt structure problem inside n8n.
Right now, n8n is combining everything (system instructions + dynamic context like customer name/session data + chat history) into one long string before sending it to OpenAI.
Because some part of that input changes on every request (like session info or user context), the final prompt is never exactly the same twice even if 80–90% of it is static.
And that’s the key reason your prompt caching is not kicking in properly. OpenAI caching only works when the prefix is identical byte-for-byte, and in your case that consistency is being broken by the injected dynamic block.
So in simple terms:
Your system prompt is mostly static (good for caching)
But it gets “polluted” by dynamic context and history
n8n flattens everything into one string instead of structured messages
So OpenAI treats every request as a new prompt
What this means practically
Your setup is not broken it’s just:
losing caching efficiency
due to how n8n’s AI Agent handles memory + context internally
To fix this properly, there are two directions:
Option 1 (quick improvement inside n8n)
Reduce or isolate dynamic context injection into system prompt
Keep system prompt as static as possible
Minimize changes in prefix section
Option 2 (clean production fix)
Stop relying on AI Agent’s internal prompt building
Build messages manually using structured format:
system message (static)
context message (separate role)
chat history as proper array
This fully restores caching behavior and gives predictable performance.
To make Option 1 actually work in practice: keep the system prompt field in the AI Agent 100% static - no expressions, no dynamic variables - and move all per-session context (customer name, status, working hours, etc.) into the first user message instead. You can inject that as a synthetic “user” turn at the top of the conversation array before the real chat starts. Since OpenAI’s caching is prefix-based and the system message stays byte-for-byte identical every call, the static prefix should cache reliably. The trade-off is that the context window structure shifts slightly, but for an 8-9k static system prompt that’s usually the most cost-effective path without giving up the AI Agent’s built-in tool execution.
I would treat this as a message-assembly problem, not a model-selection problem.
The diagnostic I would run is:
capture two raw model payloads with memory enabled;
capture two raw model payloads with memory disabled;
compare only the first 1-2k tokens of each payload;
compare cache_creation_input_tokens and cache_read_input_tokens for the same calls.
If the first changing token appears before or inside your 8-9k static instructions, the cache will miss even if most of the prompt feels “static” to a human.
The practical split is:
keep the Agent node if built-in tool orchestration matters more than cache efficiency;
route the model call through a lower-level HTTP/Responses call if you need strict message ordering and stable-prefix caching;
keep the long policy/system block byte-for-byte static;
move customer/session/history context after the stable prefix, not before it.
The key test is not whether the prompt text looks mostly the same. It is whether the prefix is identical before the first dynamic token.