I am trying to run the module ‘google-news-scraper’ in a code node but only managing to get network time-outs (after 3 mins) and struggling to find the source of the problem. I would appreciate anyones help in fault finding. Setup as follows:
const googleNewsScraper = require('google-news-scraper');
async function scrapePage() {
try {
// Await the results from the scraper function
const articles = await googleNewsScraper({ searchTerm: "gardening", prettyURLs: false, logLevel: "info" });
$input.item[0].json = { articles }; // Directly set articles as the json value
console.log("All OK");
return $input.item[0];
} catch (error) {
console.error("An error occurred:", error);
throw error;
}
}
return scrapePage();
error received:
Network.enable timed out. Increase the 'protocolTimeout' setting in launch/connect calls for a higher timeout if needed. null
I have created a Docker based on @marcus 's github page. I only made a few changes like increasing the Alpine Node version to 18 and using n8n:1.61.0 (latest).
Dockerfile as follows:
FROM node:18-alpine
ARG N8N_VERSION
RUN if [ -z "$N8N_VERSION" ] ; then echo "The N8N_VERSION argument is missing!" ; exit 1; fi
# Update everything and install needed dependencies
RUN apk add --update graphicsmagick tzdata git tini su-exec
# # Set a custom user to not have n8n run as root
USER root
# Install n8n and the also temporary all the packages
# it needs to build it correctly.
RUN apk --update add --virtual build-dependencies python3 build-base ca-certificates && \
npm_config_user=root npm install -g full-icu n8n@${N8N_VERSION} && \
apk del build-dependencies \
&& rm -rf /root /tmp/* /var/cache/apk/* && mkdir /root;
# Installs latest Chromium (100) package.
RUN apk add --no-cache \
chromium \
nss \
freetype \
harfbuzz \
ttf-freefont \
yarn
# Tell Puppeteer to skip installing Chrome. We'll be using the installed package.
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true \
PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium-browser
# Install n8n-nodes-puppeteer
RUN cd /usr/local/lib/node_modules/n8n && npm install n8n-nodes-puppeteer google-news-scraper
# Install fonts
RUN apk --no-cache add --virtual fonts msttcorefonts-installer fontconfig && \
update-ms-fonts && \
fc-cache -f && \
apk del fonts && \
find /usr/share/fonts/truetype/msttcorefonts/ -type l -exec unlink {} \; \
&& rm -rf /root /tmp/* /var/cache/apk/* && mkdir /root
ENV NODE_ICU_DATA=/usr/local/lib/node_modules/full-icu
WORKDIR /data
COPY docker-entrypoint.sh /docker-entrypoint.sh
ENTRYPOINT ["tini", "--", "/docker-entrypoint.sh"]
EXPOSE 5678/tcp
The container build and startup looks fine.
I can see that puppeteer and google-news-scraper have been installed locally.
My docker-compose file is as follows:
services:
n8n:
build: .
#image: n8nio/n8n
container_name: n8n
ports:
- "5678:5678"
volumes:
- /docker/n8n/data:/data
# - /docker/n8n/.n8n:/home/node/.n8n
#next four lines for the puppeteer build
privileged: true
shm_size: "1gb"
environment:
- N8N_BASIC_AUTH_ACTIVE=true
- N8N_BASIC_AUTH_USER=user
- N8N_BASIC_AUTH_PASSWORD=password
- N8N_SECURE_COOKIE=false
- N8N_DIAGNOSTICS_ENABLED=false
- DB_TYPE=postgresdb
- DB_POSTGRESDB_HOST=db
- DB_POSTGRESDB_PORT=5432
- DB_POSTGRESDB_DATABASE=n8n
- DB_POSTGRESDB_USER=n8n
- DB_POSTGRESDB_PASSWORD=n8n
- NODE_FUNCTION_ALLOW_EXTERNAL=axios,qs,google-news-scraper
- NODE_FUNCTION_ALLOW_BUILTIN=*
- EXECUTIONS_DATA_SAVE_ON_ERROR=all
- EXECUTIONS_DATA_SAVE_ON_SUCCESS=all
- EXECUTIONS_DATA_SAVE_ON_PROGRESS=false
depends_on:
- db
restart: unless-stopped
db:
image: postgres:12
container_name: n8n_db
volumes:
- db-data:/var/lib/postgresql/data
environment:
- POSTGRES_USER=n8n
- POSTGRES_PASSWORD=n8n
- POSTGRES_DB=n8n
restart: unless-stopped
volumes:
db-data:
Finally, after the node times out, I can see that my docker host is maxed out on cpu and multiple (10-15) chromium processes are still running.
- n8n version: 1.61.0
- Database (default: SQLite): Postgres
- n8n EXECUTIONS_PROCESS setting (default: own, main):
- Running n8n via (Docker, npm, n8n cloud, desktop app): Docker
- Operating system: Ubuntu Host / Alpine Docker
Thanks for any help in advance.