The plan we’re on has a 20 concurrency limit. I’m generating PDFs and then uploading them to Google Drive. Once they are all uploaded, a python script will run to download them to my computer for emailing out.
We’ll have much more than 20 pdfs that will be generated at one time, so the workflow will be running for an hour or more before all the pdfs are in Google Drive. I see the “wait” node but I don’t know the exact amount of time it’ll be, and I don’t want to estimate incorrectly, especially as conditions change.
Is there a way to have the workflow finish the entire queue before moving on to the next node?
As far as i understand from your question, n8n already does what you want
Most of the nodes will finish their queues before moving to the next node.
If that is not the case for you, you can always loop through the items and then connect your next node to the “Done” part of the Loop.
Also, i would separate the concerns and create a workflow for each. The workflow you described, lets say it is the Generation Workflow is a good example, it has a distinct job to generate and upload. If you want to add a new feature to the flow, like an AI Editor for your pdfs, i think it is always better to create a new workflow Editor and call it at the end of your Generator workflow. You can do this with an Execute Sub-Workflow node.
This is the current workflow, I want to make sure that all the drive uploads are done before the python script runs. Each PDF generation and upload is it’s own webhook call and workflow operation, so there will be 3k calls to the webhook and it will process and upload them. Once they are all uploaded, then I want the python script to run. What would I need to change then here to make sure that happens? Make a second workflow for the python script that activates once the upload one has totally finished? How will it know?
Yes, in this set up Google Drive Node will upload all before moving in the next node.
You can use an Execute Sub-Workflow node after the upload to call the python magic workflow.
I just made myself a download version, the google sheet has 12 urls, feeds in to drive node, drive downloads all 12 and then moves on to the next node.
