Hey everyone! Hope you are doing great
I wanted to share my use case since I’m using a self-hosted ECS architecture. Just read that filesystem configuration is not available in queue mode and S3 option it’s only available for enterprise plan. So just wondering how to deal with large binary data since I read that the only option that I have is to rely on the internal processing by using the cpu and memory which I think could not be the best option for large scale environments. Any tip or experience regarding this will be very appreciated!
Interesting! The docs does say that n8n doesn’t support filesystem mode when in queue mode at all - so that would mean it wouldn’t really help to get on enterprise plan for the s3 option? Would be good to get some clarification from the team for this.
To answer your question, seems like delegating a lot of the heavy lifting outside of n8n seems to the only way. eg. rather than passing around binary data between nodes, it’ll be passing around URIs instead and letting those external workers/apps handle the files directly.
Hey @Jim_Le first of all thanks a lot for taking the time to think about this!
I think would be nice to have the external storage enabled for community edition so I’m agree with you that the only feasible solution so far should be avoid large binaries going through nodes and store it externally in some way as you mentioned. I recently was doing some benchmarking for binary data and definitely it’s a process that can consume many hardware resources so will be interesting to find a good balance between performance and reasonable price !
!
!