Parse text input

Questions
,
1

Hello , im trying to parse this input to extract unique subdomains from it

[ { “data”: “http://www.hackerone.com:80/\nhttps://www.hackerone.com/“,\nhttps://www.hackerone.com/”,“alts”:[],“httpStatus”:200\nhttp://hackerone.com/$handle/terms\nhttp://hackerone.com/$handle/terms_acceptance_data.csv\nhttps://sub3.hackerone.com/${e.handle}`;case\nhttps://hackerone.com/${et.hackerone_username}`\nhttp://hackerone.com/%25evil.com\nhttps://sub2.hackerone.com/ \nhttps://sub1.hackerone.com/%20%20\nhttps://hackerone.com/%20%20////////%20%20%20Twitter:%20@Paresh_parmar1%20fb:%20fb.com/paresh.parmar1993%20%20-%20https://blog.pareshparmar.com” } ]

i want the output to only get the output to get the
“www.hackerone.com sub1.hackerone.com sub2.hackerone.com sub3.hackerone.com”

2

@0xSphinx easiest is a Code node, regex the host out of each url then dedupe with a Set. drop this in:

const text = $input.first().json.data;
const hosts = [...text.matchAll(/https?:\/\/([^\/\s:"'`]+)/g)].map(m => m[1]);
return [...new Set(hosts)].map(h => ({ json: { subdomain: h } }));

that grabs whatever sits between :// and the next slash, colon, space or quote, and the Set drops the duplicates. youll get www.hackerone.com and sub1/sub2/sub3.hackerone.com (plus bare hackerone.com). if you only want the ones with a subdomain prefix and not the apex, add .filter(h => h.split(‘.’).length > 2) before the map.

3

Hey @0xSphinx, regex is the right approach. One thing worth adding, your raw data has a few malformed/junk entries mixed in (e.g. %00%25evil.com, the ${e.handle} template string, and that trailing Twitter/blog text appended to one URL). The regex will mostly skip those since they don’t match a clean ://host pattern, but it’s worth double-checking your output array after running it, since look-alike strings like evil.com could sneak in if the format shifts slightly.
If you want to be extra safe, you can add a basic domain validation filter after the dedupe:
.filter(h => /^[a-z0-9.-]+\.[a-z]{2,}$/i.test(h))
This drops anything that isn’t a clean domain-looking string, which is useful when scraping data from messy/untrusted sources like this.

4

Thanks @achamm and @sergeys thats really helpful!