HTTP Request Javascript

Questions
1

Hello,

Is there a way to have javascript enabled for the HTTP Request node because the page i need to access/scrape needs javascript to function

Thank you in advance

2

So this would basically be Selenium Based scraping

3

This is not possible @Damian_K.

Http request only gets body contents without loading external resources or enabling js.

4

Hey @Damian_K!

Welcome to the community! :slightly_smiling_face:

Sadly this is not possible. However, if you’re workflow for scraping is not complex you can use the HTTP Request node to fetch the data, and then use the HTML Extract node to filter out the data.

1 Like
5

I found out that the data needed is either loaded with JS into a HTML Table and/or stored as a Javascript JSON Property

Suchh as var ticketData = {"rows": [

And i’ve not seen an option yet to extract that with the HTML Extracter

6

If the data that you is returned is in the JSON format then you can use the Function node to filter it out. The HTML Extract node will only filter the data if it is returned in HTML.

7

Allright so i’m gonna have to filter out that specific part of the page with javascript using the function node if i understand it correctly

8

This is how the flow can be: HTTP Request node (to make the HTTP Request) → Function node (to extract the required data).

9

The problem is, The Data returned is not JSON, It’s JSON inside a javascript variable in the source code of a page

10

Can you share the workflow or the output? If it contains sensitive information, you can DM me instead.

11

I cant seem to find the button to pm you here

12

Hey @Damian_K!

I tried a few solutions and this is what I have come up with. I am using the HTML Extract node to get all the script tags. Then I am using the Function node to only return the information that I need. You can use this workflow and build on top of it.

I hope this helps :slightly_smiling_face:

13

Have you tried it out? because it doesnt work for me as in, the function node does not only return the ticketdata and thus it does not see JSON

14

Allright, So i got the working regex for exactly extracting what i need

/(?<=var ticketData = )(.+]})/

Yet i get this:
Entries exist but they do not contain any JSON data.

16

Hey @Damian_K!

I tried the regular expression you provided and it gives no result. I tried logging the output and I got null. Need to investigate the regular expression and extract only the data you need.

17

I got the solution, Function 1:

let data

items[0].json.script.forEach(e => {
    if (e === null) return

    const matches = e.match(/(?:var ticketData = )(.*]})/s)

    if (matches && matches[1]) data = JSON.parse(matches[1]).rows
})

return [{ json: data }]

Function 2:

const newItems = [];
for (const item of items[0].json) {
  newItems.push({json: item});
}
return newItems;
18

I know that this is very old topic, but maybe You have a solution to check site if java script or header have redirect function :smiley:

19

Hey @Krystianroza,

We tend to prefer new topics rather than reviving a 2 year old thread :slight_smile: Using the HTTP Request node it will automatically redirect if the page is requesting a redirect but with javascript it could be trickier depending on how it is doing it. If you go to the options there is a redirect setting that could help.