I was able to find the source. It looks like a random IP owned by Chinese Tencent is highjacking the redis instance and trying to replicate the data, which then makes it read only, and then the entire n8n server freezes.
I used the following code to get timestamped ordered logs:
docker-compose logs -t --no-color | sort -u -k 3 | less > logs.txt
And then I scrolled for a few minutes until the first read only error, and went up a bit from it to see what happened just before.
Took me days to figure it out. But now that I figured it out it makes so much sense why it was super random and no pattern to the issue.
Fix implemented:
Changing the redis container to the following code to disallow remote connections:
redis:
image: redis
restart: always
ports:
- “127.0.0.1:6379:6379”