Ho to run data "through" LLM but using tools instead of exposing LLM to data

[DuggyDoo]

Hi,

I’m busy building a workflow that will at its heart have an LLM that will use OpenAI/other api. The data is rather sensitive (and large) and for this to be viable I cannot send it to the API for analysis (also use up any limits on the api). I cannot deploy a local LLM through Ollama as the machines I have aren’t strong enough.

We deal a lot with transaction data from clients. We want to run analysis on this data but give the client the freedom to query the data and not have a fixed report that we design that might not make sense to them, is not impactful enough and there is a sense of ownership when they design/query it.

The idea was to use the N8N workflow to give them a chat interface to query their data.

We were thinking of developing python driven analysis and building it over time, so store performance functions/tools. Then when the customer queries for store performance instead of the raw data going to OpenAI, a tool is called and the data analyzed locally, result anonymized and then the LLM maybe translates the results to simple English paragraphs.

Does this make sense? Or am I imagining something not possible?
If I have the LLM understand the tools at its disposal, call the tool does the raw data then actually flow through the LLM?

(I know there is debate on whether OpenAI and the like retain your data/train on your data, but irrespective clients wont agree).

Much appreciated.

[simon.lewis]

Hey DuggyDoo,

Yeah, this is a hot topic for us too. If you absolutely cannot send data over the cloud then your only choice really is to run it locally - probably means Ollama although Docker Desktop can apparently do this too (not tried it).

If you can send to the cloud but want to ensure data is not used for training or otherwise analysed / abused then choices we have played with include:

(1) OpenAI - check out their privacy statements. If you are on a paid plan / credits then (they say) the API does not use your data for training etc. We use this option (although we do not send client-confidential data like financial information). They only ‘use’ your data if you use their free options.

(2) IBM WatsonX.ai - a cloud service that has good privacy and secure connection options. There is no n8n Node for interaction with the service so you have to use the HTTP Request node and related authentication. They have a limited set of models available (IBM Granite, Llama and Mistral - see the details here: Foundation Models - IBM watsonx.ai.)

(3) Google Gemini / Gemma - needs checking to see if they operate securely / privately with your data but I think so under certain conditions - Gemini Apps Privacy Hub - Gemini Apps Help

Hope that helps.

S