I’ve got a business logic to implement.
Receiving a webhook, data sanitation, data transformation, splitting up that data and reading / inserting data from / into 10 different tables, linking related records in the tables, updating data.
This is my main workflow.
It’s calling 7 sub workflows and I’m still working on sub-workflow number 8 & 9.
It seems, as if things are more complex now.
Variables need to be passed on from one subworkflow to the next.
Error handling is complex.
More difficult to keep an overview of many SQL statements etc.
My browser spins up my CPU fan when working in the main workflow (not a technical issue - but annoying )
Are there any best practices if things get bigger?
Or my favourite: Could I use n8n for the connections (amazingly easy) and Python in the background to build the business logic?
How would debugging work in a comfortable way, if n8n is calling a Python script within the n8n docker container?
Hi there. The general question is somewhat vague, so it’s difficult to answer.
Maybe there is a guide i.e. “n8n cookbook for complex automations”, but I didn’t see anything like this so far.
Looking at your screenshot I think calling subworkflows is a good strategy to isolate complex logic inside a “big” workflow. As of using Python - this is totally possible. I mostly work with R language (a better one for data management / ETL, especially when it comes to statistics), check out my blog article.
You can run Python scripts the same way. To my knowledge Python is already built in into many linux distros, so you don’t even need to install it. However, adding some packages via pip may be needed (similar approach compared to installing packages from R’s CRAN repository).
Hi. Sorry, didn’t see your message earlier. Please reply to a message, then I can see a notification.
Normally I develop R scripts locally and then upload them to a server. So far, I didn’t use very complex R scripts in n8n, so debugging was fairly straightforward.
As of faulty data, I think if there is no data available, it is easier just not to run R at all. If the data is present, then I check it in R. It’s much easier to write checks and error handlers in R instead of looking for every possible data issue in n8n.
I’ve tried around some more and finally finished another n8n “Software”.
It worked quite well for a few records (up to 30 or even 50) - then I had to load a few thousand records and the first 20 took around 12 hours.
So I’ve given up and build the whole process in Python.
I really like n8n but I used it for the wrong use case as it seems. I’ll come back to it, once it works or if I have a fitting use case.
Not sure what you are doing exactly, but a few thousand records should not take that long.
I suspect there are some things you can do to improve the performance.
There are around 100 nodes in the Workflow and a lot of DB requests, ca. 4 integrations in API.
Probably there are some ways to optimize, but I’ve got the feeling, that I’m more flexible / better off with Python.
Yeah, n8n is not for everyone. If you are very familiar with python, that will most likely be faster.
I guess for this workflow with 100 nodes you are not splitting them up into sub workflows, this can help.
Also the structure with 100 nodes can probably be more efficient as well, I have seen some nice spaghetti in n8n. Looks nice, but doesn’t perform so nicely.
Not sure about Python as I have yet to use it. But generally in programming, it is good to not have one large method/function. You can see the same with n8n, having one large workflow simply doesn’t work so well most of the time. Especially for maintainability etc.
)