REDHAT - Docker pull for amd64 issue of latest and 0.213.0 no manifest, but its ok for old releases?

0101binary0101 · January 30, 2023, 3:48pm

Describe the issue/error/question

Docker Pull Latest tag n8n doesn’t appear to be working on amd64 arch … not sure why , it works for my arm64 system.

What is the error message (if any)?

# uname -m
x86_64
# docker pull n8nio/n8n:0.213.0
Trying to pull repository registry.access.redhat.com/n8nio/n8n ... 
Pulling repository registry.access.redhat.com/n8nio/n8n
Trying to pull repository registry.redhat.io/n8nio/n8n ... 
Trying to pull repository docker.io/n8nio/n8n ... 
Trying to pull repository docker.io/n8nio/n8n ... 
manifest for docker.io/n8nio/n8n:0.213.0 not found
# docker pull n8nio/n8n:latest
Trying to pull repository registry.access.redhat.com/n8nio/n8n ... 
Pulling repository registry.access.redhat.com/n8nio/n8n
Trying to pull repository registry.redhat.io/n8nio/n8n ... 
Trying to pull repository docker.io/n8nio/n8n ... 
Trying to pull repository docker.io/n8nio/n8n ... 
manifest for docker.io/n8nio/n8n:latest not found
#

But if I remove my old 0.209.4 image… I can pull that without issues

# uname -m
x86_64
# docker pull n8nio/n8n:0.209.4
Trying to pull repository registry.access.redhat.com/n8nio/n8n ... 
Pulling repository registry.access.redhat.com/n8nio/n8n
Trying to pull repository registry.redhat.io/n8nio/n8n ... 
Trying to pull repository docker.io/n8nio/n8n ... 
0.209.4: Pulling from docker.io/n8nio/n8n
213ec9aee27d: Already exists 
43e6571198fa: Already exists 
985418a47cca: Already exists 
371c3ce4a723: Already exists 
4f4fb700ef54: Already exists 
e5d5ff55d172: Already exists 
9395451721ce: Already exists 
c082d4370351: Pull complete 
74a9900e6e64: Pull complete 
7df3a2a0e446: Pull complete 
ab1b0abf252b: Pull complete 
Digest: sha256:ac8d9499b5cc24f272d4bab8ac60a526c328d31b8e08744ae9caafdcf2c0129e
Status: Downloaded newer image for docker.io/n8nio/n8n:0.209.4
[#

Please share the workflow

(Select the nodes and use the keyboard shortcuts CMD+C/CTRL+C and CMD+V/CTRL+V to copy and paste the workflow respectively)

Share the output returned by the last node

Information on your n8n setup

n8n version:
Database you’re using (default: SQLite):
Running n8n with the execution process [own(default), main]:
Running n8n via [Docker, npm, n8n.cloud, desktop app]: DOCKER

0101binary0101 · January 30, 2023, 4:07pm

Seems to be related to my redhat platform oddly

It works from my debian platform

[email protected]:~# uname -mr
5.10.0-19-amd64 x86_64
[email protected]:~#  docker pull n8nio/n8n:latest
latest: Pulling from n8nio/n8n
8921db27df28: Pull complete 
4e08f9bea56d: Pull complete 
b8243c93ff91: Pull complete 
2ed4f044afde: Pull complete 
4f4fb700ef54: Pull complete 
4d90deb8eda4: Pull complete 
48090d8a520c: Pull complete 
31da566294ed: Pull complete 
e98d0d5f5645: Pull complete 
e8764a9b22dc: Pull complete 
7e41daa047a1: Pull complete 
Digest: sha256:2bf42fa70e91a19115f31826a2ff1d1e17123b2ac11cf327c247da608cd962bf
Status: Downloaded newer image for n8nio/n8n:latest
docker.io/n8nio/n8n:latest
[email protected]:~#

Jon · January 30, 2023, 4:08pm

Hey @0101binary0101,

Looking at the message it looks like Redhat is using their own registry, Maybe that is the cause of the issue.

0101binary0101 · January 30, 2023, 4:09pm

Checking that out now.

0101binary0101 · January 30, 2023, 4:57pm

I’ll change the title

This is an odd problem specific to RedHat boxes … I can pull other random containers that I’ve never retrieved from docker.io but the N8N:Latest always fails… I’ve tried two other Redhat VMs same thing.

[[email protected] ~]# docker pull docker.io/n8nio/n8n:latest
Trying to pull repository docker.io/n8nio/n8n ... 
manifest for docker.io/n8nio/n8n:latest not found
[[email protected] ~]# docker pull docker.io/traefik:latest
Trying to pull repository docker.io/library/traefik ... 
latest: Pulling from docker.io/library/traefik
9621f1afde84: Pull complete 
f62b4fe8af5e: Pull complete 
ba654a766834: Pull complete 
55d4785f2007: Pull complete 
Digest: sha256:bb7be8d50edf73d8d3a812ac8873ef354a0fe9b40d7f3880747b43a3525855d2
Status: Downloaded newer image for docker.io/traefik:latest
[[email protected] ~]# uname -mr
3.10.0-957.el7.x86_64 x86_64
[[email protected] ~]#

Jon · January 30, 2023, 5:12pm

Hey @0101binary0101,

Sadly I don’t think there is anything we can do, The image is available but I don’t know how often RedHat update their own registry if at all. Maybe they don’t see us as being important enough to include or maybe they have a submission process.

It could be worth checking if you can add to the registry or configure docker to also include other registries.

0101binary0101 · January 30, 2023, 5:13pm

@Jon

Could it because N8N has got some ‘Unknown’ Arch models listed , I can pull n8n/base:18 but not n8n/base:16

Jon · January 30, 2023, 5:18pm

Hey @0101binary0101,

Shouldn’t be that, The error you get is the same one I see if I use a tag that doesn’t exist. At the moment the best guess is that redhat just don’t include it for some reason I am basing this on the image existing (you can see it on the website) and it works from other Linux distros.

It would be nice to work out what their process is but we do also have the image on ghcr.io so maybe that one would work instead.

0101binary0101 · January 30, 2023, 5:35pm

Redhat has a registry configuration file I made sure it had docker.io and ghcr.io listed.

[registries.search]
registries = ['registry.access.redhat.com','docker.io','ghcr.io']

Going by the commands on: https://github.com/n8n-io/n8n/pkgs/container/n8n

Even that’s failing - really strange

[[email protected] ~]# docker pull ghcr.io/n8n-io/n8n:[email protected]:2a608c9da1a7dbb094f2ef31b7f93d6ee6add99592942f421355d7033fab6f49
Trying to pull repository ghcr.io/n8n-io/n8n ... 
manifest for ghcr.io/n8n-io/[email protected]:2a608c9da1a7dbb094f2ef31b7f93d6ee6add99592942f421355d7033fab6f49 not found
[[email protected] ~]#

I might wait to see if any other Redhat users also have the same oddity.

Jon · January 30, 2023, 5:42pm

Just incase I have missed something… @netroy do you have any ideas?

0101binary0101 · January 30, 2023, 11:30pm

For now I’ve modified my automation on my Redhat box to pull via my Debian host, it works albeit a bit clunky with the docker save, scp and docker load then re-tagging to include docker.io/n8nio/n8n:latest tagging

It’s just a bit strange why it was quite happy working daily until the 23rd Jan in the evening check.

netroy · January 31, 2023, 10:19am

I tried running docker pull ghcr.io/n8n-io/n8n:[email protected]:2a608c9da1a7dbb094f2ef31b7f93d6ee6add99592942f421355d7033fab6f49 on two different machines running debian and arch, and did not face any issues.
Tried the same in a Fedora 35 VM, and still can’t reproduce this.

@0101binary0101 is the Redhat machine by any chance behind a proxy, or on a VPN?

0101binary0101 · January 31, 2023, 11:49am

@netroy - No VPN or Proxy involved, just firewalls

RH Version of docker is old (it came on the RHEL Extras) maybe there’s something quirky there:

[[email protected] ~]# docker --version
Docker version 1.13.1, build 7d71120/1.13.1
[[email protected] ~]#

What makes no-sense is the following on Redhat (I’ve tried 2 x RH 7.9 boxes, 1 x RH 7.6 box):

Can pull other non-n8n containers … that have been published within the last and few days
Can pull n8nio/n8n:0.212.0, and others releases prior to that.
Can pull docker.io/n8nio/base:18,
Cannot pull docker.io/n8nio/base:16

[[email protected] ~]# docker pull docker.io/n8nio/base:18
Trying to pull repository docker.io/n8nio/base ... 
18: Pulling from docker.io/n8nio/base
213ec9aee27d: Already exists 
f379b689aea3: Pull complete 
fe299d5780c0: Pull complete 
c34a027bbf26: Pull complete 
4f4fb700ef54: Pull complete 
217dbe36c351: Pull complete 
2a057e06f51c: Pull complete 
Digest: sha256:ccfe42bcb61f9f59d39134ee602a5287739d982cac0df48095042523eef219f2
Status: Downloaded newer image for docker.io/n8nio/base:18
[[email protected] ~]# docker pull docker.io/n8nio/base:16
Trying to pull repository docker.io/n8nio/base ... 
manifest for docker.io/n8nio/base:16 not found
[[email protected] ~]# docker pull docker.io/n8nio/base:16
Trying to pull repository docker.io/n8nio/base ... 
manifest for docker.io/n8nio/base:16 not found
[[email protected] ~]# docker pull docker.io/n8nio/base:16
Trying to pull repository docker.io/n8nio/base ... 
manifest for docker.io/n8nio/base:16 not found
[[email protected] ~]# docker pull docker.io/n8nio/base:16
Trying to pull repository docker.io/n8nio/base ... 
manifest for docker.io/n8nio/base:16 not found
[[email protected] ~]#

Cannot pull ghcr.io version but can pull other images from ghcr.io

[[email protected] ~]# docker pull ghcr.io/home-assistant/amd64-hassio-supervisor
Using default tag: latest
Trying to pull repository ghcr.io/home-assistant/amd64-hassio-supervisor ... 
latest: Pulling from ghcr.io/home-assistant/amd64-hassio-supervisor
ca7dd9ec2225: Already exists 
e6d365cffc05: Pull complete 
25233afa79ae: Pull complete 
3672428e239a: Pull complete 
1ad69a6e86a8: Pull complete 
abb2341072b3: Pull complete 
219e719719e5: Pull complete 
4f4fb700ef54: Pull complete 
bd60c07dab27: Pull complete 
4da8b8624247: Pull complete 
bcbf6143cce3: Pull complete 
08aeb9fd58d6: Pull complete 
aba276d9b263: Pull complete 
51992de4c1e6: Pull complete 
Digest: sha256:fee9e781c02b4442c8fe49e62d73a7c9fa0459c7d97a867dda7a493f4991cc46
Status: Downloaded newer image for ghcr.io/home-assistant/amd64-hassio-supervisor:latest
[[email protected] ~]# docker images | grep hassio
ghcr.io/home-assistant/amd64-hassio-supervisor   latest              8748f9ee667a        6 days ago          293 MB
[[email protected] ~]# docker pull ghcr.io/n8n-io/n8n:[email protected]:2a608c9da1a7dbb094f2ef31b7f93d6ee6add99592942f421355d7033fab6f49
Trying to pull repository ghcr.io/n8n-io/n8n ... 
manifest for ghcr.io/n8n-io/[email protected]:2a608c9da1a7dbb094f2ef31b7f93d6ee6add99592942f421355d7033fab6f49 not found
[[email protected] ~]# date -d "6 days ago"
Wed Jan 25 11:26:10 GMT 2023
[[email protected] ~]#

Debian x86 box on the same network works without issues:

Can pull n8n from docker.io
Can pull n8n from ghcr.io

Digest: sha256:2a608c9da1a7dbb094f2ef31b7f93d6ee6add99592942f421355d7033fab6f49
Status: Downloaded newer image for ghcr.io/n8n-io/[email protected]:2a608c9da1a7dbb094f2ef31b7f93d6ee6add99592942f421355d7033fab6f49
ghcr.io/n8n-io/n8n:[email protected]:2a608c9da1a7dbb094f2ef31b7f93d6ee6add99592942f421355d7033fab6f49
[email protected]<squawk>-deb-01:~# docker images | grep n8n
ghcr.io/n8n-io/n8n                                <none>      78b7bdb97cd8   3 days ago      1.86GB
n8nio/n8n                                         latest      b874969728da   3 days ago      595MB
n8nio/n8n                                         0.212.0     68ef35de594b   11 days ago     582MB

I’m going to build another RHEL VM (without some security mods) box.

0101binary0101 · January 31, 2023, 12:49pm

Ok it’s resolved, it was the docker version that is released as part of the Redhat Extras CD,

OLD version:

[[email protected] ~]# docker --version
Docker version 1.13.1, build 7d71120/1.13.1
[[email protected] ~]#

Newer Version:

[[email protected] ~]# docker --version
Docker version 20.10.23, build 7155243
[[email protected] ~]#

something changed in n8n releases between 0.212.0 and 0.213.0 which causes a quirky issue with the older Redhat Docker Extras release.

I’ve installed a new release of docker via the centos source (reference: Install Docker Engine on CentOS | Docker Documentation) and can now pull n8nio/base:16

[[email protected] ~]# docker pull n8nio/base:16
16: Pulling from n8nio/base
8921db27df28: Pull complete 
4e08f9bea56d: Pull complete 
b8243c93ff91: Pull complete 
2ed4f044afde: Pull complete 
4f4fb700ef54: Pull complete 
4d90deb8eda4: Pull complete 
48090d8a520c: Pull complete 
Digest: sha256:5ed0cee768089444ab0f77efb75323b5455185c542253a8b885a8c16b926fc9b
Status: Downloaded newer image for n8nio/base:16
docker.io/n8nio/base:16
[[email protected] ~]# docker --version
Docker version 20.10.23, build 7155243
[[email protected] ~]#

Thanks for bearing with me on this.

system · February 7, 2023, 12:49pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.