Sharing my first "hard" (working) workflow as a dev - Recursively get all sub-folders from a google drive folder

Built with n8n
,
1

Hi,

I am a dev new to n8n and here is my first workflow to solve one problem : the google drive api does not recursively give you all subfolders, you need to orchestrate several api calls to have all subfolders.

This workflow lets you make a webhook call with the id, and you get a list of all subfolders ids.

I have two objectives with this post:

  1. share it since it seems to be a shared problem not solved in n8n after a few searches
  2. Understand what I did wrong designing my workflow because this was so hard to implement and took so much time when it is easy to do in pure code.

Here is the workflow

I hope it helps someone in need but also here are my observations/questions coming from a dev experience:

  1. The only reason I did not do this a a single code node is because I lack the ability to import the google drive credentials from n8n in the code node. Very frustrating. So I had to use the google drive node and “play” around it.
  2. I would love to know all the things I did there that are not in the spirit of n8n or that I could have done more easily. I think my solution is too “convoluted”.

Thanks a lot for any remarks, I think I must be doing something wrong for it to be so hard to do something so simple :sob:.

3 Likes
2

Hi @n8ncmat Welcome to n8n :n8n: community :tada:

This is an awesome first contribution, thanks so much for sharing it!

The workflow looks super advanced…
I think I’ll need to take some time to fully understand it, but it’s clear you’ve put a lot of thought into solving a real problem :pray:t2::clap:t2:

3

Thanks a lot, man! This saved me a shit ton of time.

1 Like
4

Amazing nice work :slight_smile: I guess quite a few hours later :slight_smile: thanks for sharing, you get use to it, so many ways to achieve the same thing in n8n, I find keep updates and it will just become natural :slight_smile:

5

I had issue with the loop while trying to use it with the Execute another workflow node.
I came with this solution, if it can help

1 Like
6

UPDATE in case a few people are thinking about using this.

I have been struggling with memory issues (using n8n cloud you get very low memory) with this workflow.

To go around this, here is the updated logic I used:

First, set up a database (postgres on supabase for example) table. Most basic implementation will have 4 columns : id(primary key, automatic), job_id (text), folder_id (text), processed (boolean)

You will have two workflows.

Workflow 1: the orchestrator

It takes the folder_id you want to have a list of all subfolders from, generates a job_id and insert a row in DB with processed=False, the folder_id and the job_id.
Then it will make a request to see if there are one or more rows with processed=False for our specific job_id. If there is not, it can make another request to get the list of all uniques folder_id for our job_id which is the final information we want. If there is, launch the second workflow (the worker). Make it a loop with a wait until there is no more processed=False.

Workflow 2: the worker

Takes a job_id as input. Take one row that has processed=False. Make a request with gdrive node to get all immediate subfolders. If zero, simply change the row to processed=True. Else, insert all folders found - each one is a row - with processed=False and our job_id, then set processed=True to the initial row we processed.

I can’t directly share the workflow but don’t hesitate to recreate it with this logic.

This will allow you to minimize the memory footprint, as the orchestrator workflow will have minimal memory use (does not make a gdrive call), and we only do one gdrive call at a time in our sequential use of the worker workflow.

Disadvantage : it takes much more time. You can (and I did but it’s a little more advanced) introduce parallelism in the worker workflow (process all rows that are not yet processed, not only one).

Hope this helps