Workflow constants/parameters (Local Data Store)

Hi

I think it would be useful to be able to set some constants/parameters on the workflow.

Example 1: credentials
When working with Airtable nodes, they all use the same credentials. So, I could set on the workflow that all Airtable nodes use the same set of credentials. This could be then overwritten on the node level, e.g. one particular node specifies a different set of credentials.

Example 2: parameter
When working with Airtable, I must specify the id of the Airtable Base to interact with. In many cases the base is the same across all nodes. Currently this can be solved by either: using expressions to reference the base id from one node, or using a Set node which is not connected to the flow.

Thank you for your consideration!

I was thinking the same thing this morning when working on a new workflow. It would be nice to have something in the UI for a workflow where I can set a variable and if needed update it later.

I did notice there was getWorkflowStaticData but it seems to have some limitations. Moving that sort of functionality to a “workflow settings” prompt would be handy and for bonus points hook it into the new user features so I can set which users can update and delete workflow level variables.

Thanks for the idea @theneilkirk this sounds like a neat feature :bulb:

You mention that “all Airtable nodes use the same credential” - I wonder if it might be a bit more flexible to define these cred/ param variables at the wf level and then have an easy way to reference them within node parameters. This way, a large number of Airtable nodes could share a “linked” cred but wouldn’t be a hard rule for every Airtable node in the flow.

Do you see any downside to this approach?

@Jon user management will have access roles for individual workflows. At first, there will be a handful of set roles (similar to Google Docs sharing) - we are however architecting the backend in a way to allow more flexibility further down our roadmap.

2 Likes

That exactly what I was trying to describe @maxT. Being able to pick a default set of credentials for each node (that expects them) in the “workflow settings” or whatever, like @Jon said. Each node could then have a different set of credentials set on the node level :star_struck:

And a similar thing for arbitrary key-value pairs. Like I could set a “workflow parameter” to:
"airtableBaseId":"[AIRTABLE BASE ID]" and then I can reference that as an expression in the Airtable Nodes where base id is expected. As @Jon mentioned again, I think you could do this with getWorkflowStaticData but it would require you to use a Function node simply to set them at the start of the workflow. Having the ability to set them on the workflow level would be neat.

3 Likes

Hey @maxT. Any further thoughts on the idea of having workflow parameters/variables? Like example 2 above?

Some use cases:

  • when I use Airtable, I need to input the base id in each and every node. And if it needs to change, it’s every node…
  • when I use the HTTP request to call an API, I use the same URL prefix in as many nodes as I use for the API. Can easily be 4+ times per workflow

Currently, create a Set node and then reference it’s parameters, like this:

1 Like

@theneilkirk on our immediate roadmap we do have a local data store planned. It’s V1 scope would still need to be determined during the research phase of the feature. However, at minimum I expect it to be a key:value data store with a lightweight user interface, hosted on your instance. I’d also expect that you’d be able to reference this data in expressions (likely picking data from the left variables tree).

So I see this as the current solution to your usecase above, since you could store arbitrary key:value pairs in the data store (things you typically set in the Set node) and reference across workflows. This would also afford you a single source of truth across workflows for this type of data. Seemingly, we’d just need to ensure there is expressions support on the credential parameter in a node, so it could be specified via name (or ID) with expressions.

Does that sound like it would solve your solution?

1 Like

That sounds amazing!

Is there a ticket or similar I can follow?

Glad to hear that it sounds like this feature would be helpful! Unfortunately, there isn’t a public facing ticket to track progress at the moment.

I also think that some kind of workflow configuration is very useful. I made up my mind what my current use cases are.

Thinking about different structures like key/value pairs, one-dimensional mappings (e.g. language → parameter), multi-dimensional mappings (e.g. translations), lists, or complex object structures I came across the staticData. That with its global scope would be exactly fitting my needs. But why the hell isn’t it accessible within manual executions? In this way it is totally useless for using it for configurations.

So I am thinking of putting my config to an OpenSearch instance and retrieving it at the beginning of a workflow. But I would have to duplicate this request for each trigger and the addressing of the firing configuration node would be killing everything in complex workflows.

Starting with a key/value configuration would be a great first step. But I think complex data structures with the possibility to be requested with JMESPath would be awesome.

2 Likes

I agree w all above here. A “workflow-context” set of variables that could be called inside any node and could be easily updated would be ideal. When I have this need I currently use “node-red” instead of n8n.

Good to know @atsmusz thanks! Will keep these points in mind when discussing Local Data Store feature internally. Maybe there is a way to get a 2-for-1 where the functionality of that feature also powers a workflow level data store (conceptually, there’s a lot of overlap; just really about contexts).
For your specific case @atsmusz have you tried having a Set node early in your flow where you have various config params set, that you then reference later in your flow? Is a workaround but have found various users doing this and for their style of working wasn’t considered “hacky”. Let me know!

Yes I have done that but it’s not exactly as “friendly” a method as it could be. :smile:

Additionally (and maybe I’m doing something wrong here) if I have a “loop” process, that reference works for the 1st iteration but then no more.

There does not seem to be a way to keep that data accessible?

I hear you there on it not being exactly a native feature :slight_smile: So it is possible to directly reference a specific item of data on a node.
I’m on my phone but from memory:

  1. Open the node, and switch to JSON view
  2. Click on the specific key:Val you want to reference
  3. An orange copy button should appear in top right of the JSON view
  4. That button opens a dropdown menu, click on “Copy Item Path”. Then paste this in an expression. Now, that expression will reference item x from the set node, for every item of data sent into the node.

For reference, “Copy Parameter Path” in that same dropdown is the same as when you’re referencing from the expression editor.

We’re aware that copy button isn’t the most discoverable, and in future will find a better pattern for that interaction. Hope this helps!

2 Likes

+1 this would be a very useful feature!