Community node - complex web scraping with Puppeteer

Hi,

I have created an extended version of the existing Puppeteer node because I needed complex interactions features and I decided to share it with the community. If you try it let me know if you find a bug or need clarification on how to use it.

If you want to see it in action, I made a video here: https://twitter.com/i/status/1574320458245472256

3 Likes

Hi @hckdotng

Welcome to the community!
Always happy to see new community nodes. :slight_smile:

Have you contacted @drudge to see if you could work together on extending the existing puppeteer node? It would be very nice to see you guys working together to make one excellent node.

Hi, I’m in contact with @drudge, I hope we will have the opportunity to collaborate. I didn’t reach him for this because I had to break the compatibility with a very different logic but if he wants to modify the original node I’ll be happy to help.

1 Like

I developed a similar node to work with headless browser, that may fits your needs

https://www.npmjs.com/package/n8n-nodes-browserless

project is in the early stage so might need contributions from those who are interested in this topic.

2 Likes

this plugin works very well for me so far. https://www.npmjs.com/package/n8n-nodes-browserless

However there needs to have the ability to use a proxy, as sometimes I need to force the browser to use a specific proxy, and I think I remember a couple of spelling mistakes in some of the menus, but that wasn’t a issue for me really.

Would be good to have it do actions like “scroll to the bottom”

but I don’t want to hijack this thread.

Hello! I am finding great success with your fork! One issue I have is that I can’t figure out how to get any of the injection features working. Can you provide an example? The twitter link you provided is broken…

Also, is there any way to interact with the puppeteer process while it is running outside of the action you exposed via UI? I’m trying to find a way to save the session or cookies.

2 Likes

I just replicated the example in the node to see how it works but it returned nothing. May I ask why is it?

May i know how did you get the node running?

Hi, I didn’t do it personally, but we had to do a lot of fiddling with docker makefiles to get all the dependencies.

Thank you for the reply. So we can just use the original plugin as it to make it run? need to install a bunch of dependencies before it can really run?

could you please share an example docker “compose” file ?
thanks