I don’t know if anything like this already exists, but I am looking for a different way to pass data variables than referring to a previous node in the expression. This is for 3 reasons…
I find the current method insanely buggy. So often the data is showing in the input, but when I drag the schema into the expression, it says no data exists. I have so many flows where this has been an issue. Especially on very long flows. The data I need is in the earlier nodes, but if I refer back to them they simply don’t work. Here is a case in point:
The variable is showing as existing on the input, but even when dragging it into the expression, suddenly it becomes an error.
When running sub work flows, it can be very difficult to pass data from the main workflows trigger into the sub work flow. Again, a real issue on more complex flows.
The above example is actually a less than ideal hack I have had to create, where I have created a temp table in supabase to store the data store variable so I can access it in this sub workflow. This solution sucks for many reasons, but is the best hack I can find for now.
So often the js breaks when restructuring flows, and so you have to go back through and fix everything. This becomes very tedious on complex steups.
If there were some session mini database to store and retrieve data variables at any point in a flow or subflow, this would make life so much easier, and hopefully more stable.
Does anyone know of some existing solution to this?
@YodAI , the problem you experience can be explained by your expression relying on auto-determination of the paired items (via .item.). This could fail when the workflow includes IF nodes, switches, and custom nodes. That is when one-to-one relationship is deviated.
By selecting the right reference expression you can make your data processing in the workflow reliable, no need for a workaround with a “temp data store”.
Thanks @ihortom - I think I get what you are saying. It seems a bit of a usability issue then that it does not just auto-populate with the optimal expression when you drag the variable into the expression field. However, I don’t see how this fixes the issue with the sub-workflows. It would be far more user-friendly to have a simple variable that can be referenced or updated as you go, and where the expression never changes.
it still feels very much like both a bug (as in that the item is defined, in this example ‘var’, and it is the expression that n8n created when dragging the variable into the field, and a usability issue - I should not need to go through the docs to find ways to hack it and make it work. my feeling is, if you fixed this from a users perspective, you would get a much higher adoption and retention rate. I know a few people who have been put off n8n simply because of this field. They prefer Zapier only because this expression is essentially hidden in a user friendly interface (that just works).
I get from a coding perspective this is a non issue, but this one fix would do more to help you grow as a company than I think you realize. (I know from personal experience - I started FusionHQ over 15 years ago, which was the first ever drag and drop no code builder for marketing processes. We had similar issues that put off many users. Once resolved the difference in retention and reduction in customer support was night and day)
The actual linked items are only known at the run time. n8n might need a bit of help from you to reference the exact item you want (using first(), last(), or other methods whichever is appropriate).
It would be far more user-friendly to have a simple variable that can be referenced or updated as you go, and where the expression never changes.
There is such a variable, it’s called static data. However, its scope is the current workflow (not available from another workflow).
You are still probably misunderstanding the usage. As I mentioned, you need to use the appropriate function to access an item from previous nodes. Once it is the right one for your scenario, it is reliable and consistent.
Do you mean to reference the nodes form a different (sub-)workflow? Such variables exist but they are read-only by the workflows (their values should be set beforehand via UI).
I understand your frustration. However, it is not a bug but rather determined by the product design. When “programming” via UI (drag&drop) .item. is used by default. However, indeed, in some cases you would have to replace it with other referencing means.
Also, when comparing Zapier and n8n, keep in mind that Zapier positions itself as “no-code” where as n8n is rather “low-code” which gives much more flexibility and customization. I cannot imagine Zapier used in the workflows as complex as it could be built with n8n.
@ihortom - thanks again for a detailed breakdown. However, I would say, having built multiple SAAS companies and web app projects, it is a bug when something does not insert a functioning expression. It may not be a bug in terms of how it has been coded, but it is generating incorrect code within the user experience. It is exactly this type of issue that causes users to leave.
I do appreciate the difference between low and no code. And I do think n8n is superior to Zapier for multiple reasons. And, I am not suggesting that ability to use code in the node be removed. However, for n8n to really scale and grow to its full potential, this is a function that would make a dramatic difference.
Also, you have yet to address the real main issue, which is passing this variable to a sub-workflow. This is a major when it comes to the potential of the platform for more advanced use cases. I currently have flows that have 5 or 6 subflows working together, with large complex connections of various nodes in each. This makes data handling in the current form a headache. I am also working on building a multiskilled chat agent, and this is going to get exponentially worse.
I can hack things for sure, but this is not a preferable solution to systems design. Clean management of data variables would be a much better approach (both from a systems perspective, and as a user).
It feels like the system is so close, yet has a massive gap of missed opportunity. As you say, there are certain functions, like fixed variables, but these are only readable from a flow. Having a simple node to read and write to that function, with a variable for session ID (so each item in the fixed variables could be unique based on the session), and having this available from the input schema of every node, would solve everything.
Userability would be significantly improved, functionality expanded, and the need for creating hacks or custom code dramatically reduced.
Hey @YodAI , thank you for your detailed feedback. We really appreciate it. Your insights are valuable, and I will share them internally.
As for sharing the data between sub-workflows it is achieved in such a way that the data of interest would have to be passed on to Execute Workflow node. You cannot reference any node in the parent workflow from the sub-workflow directly. However, the sub-workflow will have all the data it needs from the parent workflow assuming you follow this implementation.
The sub-workflow’s Execute Workflow Trigger will have access to all the data that was passed over to Execute Workflow node in the parent workflow.
Similarly, the output data in the last node of the sub-workflow will automatically become available to the parent workflow. The parent workflow can continue where it was left before passing the data to the sub-workflow if needed.
In other words, the data has to be explicit when passing it over between (sub-)workflows. The explicitly is achieved by having the data to be passed over in either the node that calls a sub-workflow or in the last node of sub-workflow if that data should be available in the parent workflow.
I also do not expect any usage of the “temp data store” here, the data just need to be passed explicitly (available in the calling node or the last node of the sub-workflow).
Having said that, I am aware that some automation tools provide storage access that allows the data to be written to and read from any workflow/scenario, no matter the dependency the workflows have between each other. It is a great feature, no doubt. Hopefully it will be available in n8n at some point too. Meanwhile the alternative to that is an external storage.
@ihortom - the issue I have with sub workflows, is that it is when running them on multiple items. When you want to run a block of multiple items, each with multiple objects, in a uniform way, then the addition of the extra field breaks the formatting. They way you are suggesting works in many use cases, but not all. Using an external reference (eg supabase table) does not work either, as you need to pass a session variable from one flow to the other as a reference. But if I could do this, then I could just pass the primary variable anyway. Many of the flows became so long, that they were making the browser so slow that things became unusable. Breaking this into subflows really helps this, and modularises the design, which is great. But not so great if there is no way to have a consistent identifier that when called they are part of the same session of the original workflow.
Is there no way to reference a session ID across a single sequence (made of both master and sub flows)? If so, this would also fix the issue, as I could then use an external database as has been suggested.
