PuppeteerJS Execution (or other Web Scraping)

i tried making the n8n-puppeteer node work from github but I just cant get it to work, would love for it to just be part of n8n as a node we can call.

The good news is with the community node repo on the way the process to add external nodes will be a lot easier.

1 Like

:wave: Hey there, author of the n8n-nodes-puppeteer node. What issue were you having getting the node running?

just trying to get it working with an already running docker instead of having to go back and make a dockerfile and doing it again.

I’ve published a new version of my n8n-nodes-puppeteer node which includes the ability to Get a PDF from a URL with all options supported by page.pdf().

1 Like

I’m having a hard time using your puppeteer node on my n8n Docker version.

I installed the n8n-nodes-puppeteer v0.6.1 via Community Node but that’s not working, I assume because I’m running on Docker:

ERROR: Failed to launch the browser process! spawn /root/.n8n/nodes/node_modules/puppeteer/.local-chromium/linux-1002410/chrome-linux/chrome ENOENT

So I’ve been trying to follow the installation process for Docker users but it’s a little complicated for me, as there’s more than one way.

This is what I’ve done on my server:

mkdir n8n-puppeteer
wget https://gist.githubusercontent.com/drudge/4be1238282a5db30b3786b5de394d13d/raw/aa3103a43c1e8933105d46284ad8061a8ad91ea1/Dockerfile
wget https://gist.githubusercontent.com/drudge/4be1238282a5db30b3786b5de394d13d/raw/aa3103a43c1e8933105d46284ad8061a8ad91ea1/docker-entrypoint.sh

Got both from Running puppeteer node in n8n · GitHub

Then I run the first command you recommended (without the first “buildx” as that doesn’t work)

Not working: docker buildx build -t n8n-puppeteer --build-arg N8N_VERSION=0.186.0 .
Worked: docker build -t n8n-puppeteer --build-arg N8N_VERSION=0.186.0 .

A lot of stuff got installed, and then when I ran the second command:

docker run -it --rm --name n8n -p 5678:5678 --cap-add=SYS_ADMIN n8n-puppeteer

I get this error:

docker: Error response from daemon: driver failed programming external connectivity on endpoint n8n (02fe8415295879b699715eb2e733e8f2b7d464db7bfb9f39c63f492ae1946916): Error starting userland proxy: listen tcp 0.0.0.0:5678: bind: address already in use.

You also recommended this other link on your FAQ:

But decided to ask here first, just in case I’m doing something wrong.

I just want to run a web scrapper on my current n8n installation.

Any help would be greatly appreciated.

Thanks for your contribution to the community.

Hi there!

Are you running other n8n installation on the machine running the docker run command? Based on this error, something else is already listening on port 5678:

Error starting userland proxy: listen tcp 0.0.0.0:5678: bind: address already in use

Well, yes. My main n8n is there.

So I need to have another VPS to be able to use your node?

The docker run command creates a new container that will run a new instance of n8n. It sounds like you want to install the module in an npm hosted instance instead.

Oh boy… now that’s becoming complicated.

yeah i gave up and instead installed puppeteer on a different host, and just ssh to it in my workflow, sending dynamically created .js files to the other host to execute with puppeteer and then ssh them back into the n8n flow.

It would be nice to have puppeteer work natively in n8n but for me with docker and everything its too much hassle sadly. IF it was all bundled in one community node would be awsome. but understand there are potential issues with that.

1 Like

@drudge
Are you planning to release a manual for docker-compose?
I have installed node from community nodes but it returns me this error as in the beginning of the thread. Basically checking the path it appears that the node installation did not install the files on the host.
There is no /nodes/ in this folder.

Error: Failed to launch the browser process! spawn /home/node/.n8n/nodes/node_modules/puppeteer/.local-chromium/linux-1002410/chrome-linux/chrome ENOENT

EDIT:
I found that my files are in other path like /root/n8n/
How can i edit path where n8n lookup for files?
I’m using docker-compose.yml so i added this line under “environment”:

     - N8N_CUSTOM_EXTENSIONS="/root/n8n/.n8n/nodes/"

but it’s not working ie. error is the same.

you need to also install chrome browser as a docker to get it working and link them all together, its a right pain hence my post above.

@rafuru

If it can help you, I share you my docker-compose used to work on my extended version of the Puppeteer node: n8n puppeteer docker-compose · GitHub

PS: You don’t need the second volume in my docker-compose (it was to link the dev version of the node, but you will just have to install it from the Community nodes UI)

@hckdotng
I tested the installation of your node through the community node UI but the problem is the same.
My n8n installation is in /root/n8n/ and puppeteer installs there and when I try to fire up node it returns an error that chrome cannot be found in the path /home/node/.n8n/nodes/node_modules/puppeteer/
Which is strange to me because in general there is nothing in /home/node/.n8n so I don’t know why exactly in this path n8n is looking for chrome instead of my installation location.

EDIT:
I’m using a DATA_FOLDER variable from .env in my docker-compose and that’s why i have other n8n path. Community nodes don’t use this?

@rafuru Have you tried a fresh install with my docker-compose? It provides automatically everything needed.

@hckdotng
So even if i use your docker-compose file i still need to build n8n with dockerfile?
If i’m right reading your docker-compose file… So this will be very dissapointing info, probably most of us will not do that to use your node :frowning:

EDIT:
I installed google-chrome directly on the system and set the path however the error is still the same.

@hckdotng
Is it perhaps possible to use volume type: bind to use external chrome?

Maybe if you bind /usr/bin/chromium-browser but I think you’ll have more luck on the Puppeteer community to find someone who can help you with this specific setup.

I really think if you’re using Docker it’s easier to build an image (the docker compose provided will do it automatically on the first launch).

@hckdotng
I tried installing docker container according to your instructions here → Puppeteer and n8n. I recently needed to automate complex… | by hck.ng | Sep, 2022 | Medium however in my case n8n doesn’t even turn on :frowning:

In the logs it only repeats:
{"level": "debug", "message": "Wait tracker querying database for waiting executions", "metadata":{"file": "WaitTracker.js", "function": "getWaitingExecutions", "timestamp": "2022-10-03T19:16:22.767Z"}}

When installing I get an error that "version: “3.7” is not correct. It works for 3.3.
And i changed n8n version from 0.193.5 to 0.196.0