I have an application that needs irregular triggering, think “emulate human behavior“, while scraping websites. Ideally, the operations should not show any pattern. Unfortunately, there is no way to achieve this elegantly in n8n.
My current approach is to set the scraping times manually (using my own behavior) and once a scraping is done, the next scraping time is set. The schedule trigger is set to fire every minute and check if scraping needs to be done.
This is pretty wasteful and clogs all my logs with tiny, few-millisecond, executions. The schedule trigger cannot be reconfigured on-the-fly, meaning that the workflow needs to be deactivated/activated for new settings to have an effect.
I would like to know if there could be a more elegant solution to this. Of course, external triggering via web-hooks would work, but I’d like to keep everything within n8n if possible.
What is the error message (if any)?
None.
Please share your workflow
Information on your n8n setup
n8n version: 1.109.2
Database (default: SQLite): SQLite
n8n EXECUTIONS_PROCESS setting (default: own, main): main
Running n8n via (Docker, npm, n8n cloud, desktop app): Docker
One way that i can think about this is to make these scraping workflows to be a sub workflow
while the main workflow is basically just calling all this scaprring worfklow, but before calling it, add a wait node with a random time
something like this
this way, even though let’s say the schedule trigger is static and you set it to hourly, at least for the wait time you can randomize it in the wait node, to execute in between minute 1-60
To emulate human usage, one needs to respect complete downtime during night (humans sleep), several hours pause between calls, then perhaps a couple of intensive usage over a few minutes… It’s quite erratic.
Your proposed solution is similar to what I have done, but moves the calls into a different workflow. Although, this would indeed allow to hide the useless logs, for when there is nothing to scrape (prevent saving the logs of the schedule trigger), this still triggers every minute.
This does not translate well into the new pricing scheme, where every execution counts: an execution per minute is 1440 executions per day, however perhaps 10 executions/scrapings will actually be useful. It does not really matter to me right now, as I’m running a personal test project, but if one had a legitimate business need for this, then it would not be very efficient.
Although, I understand the reasons for the new pricing system, this particular application would be an edge case. External web-hook triggering remains the only viable solution in this case.