AWS Image Download Issue

[Will_Roske]

Hello, I am running n8n on AWS and I am trying to download loads of images from my s3 bucket. The download node will not let me download my first large test set (1,399 images) in one go, so I need to set it up in a loop.

The number of images will never be the same. The problem is that the last iteration of the loop will not have the same number of images as the rest of the runs. I tried running it without the code nodes, but then the loop over items node just buffers until it times out.

Now, I’m having the same problem, but it is buffering at the function node. I have been trying to get ChatGPT and Claude to write me code to put in these nodes so it sets all of the first batches to 200 then the last batch to how ever many images are left so it doesn’t get stopped, but it just will not work.

The one time the loop actually completed was when the function node inside the loop corrupted the images and turned them into a line of text. So, the loop finished, but it didn’t output images. I have no idea what is wrong, and I am open to all types of solutions. Here is the section of the workflow as it sits right now.

I would like to insert the workflow, but I can not figure out how to. I clicked on the </> and pasted my copied workflow, but it is just pasting code and when I post the conversation it does not come up as a set of nodes.

[Will_Roske]

{
“nodes”: [
{
“parameters”: {
“operation”: “getAll”,
“bucketName”: “REPLACE_ME_BUCKET”,
“returnAll”: true,
“options”: {}
},
“id”: “4a4fad21-06bf-429e-baf5-abcb92cd20f2”,
“name”: “List S3 Images2”,
“type”: “n8n-nodes-base.awsS3”,
“typeVersion”: 1,
“position”: [-1460, 2220],
“credentials”: {
“aws”: {
“id”: “REDACTED_CRED_ID”,
“name”: “AWS account (sanitised)”
}
}
},
{
“parameters”: {
“bucketName”: “REPLACE_ME_BUCKET”,
“fileKey”: “={{ $json.Key }}”
},
“id”: “65e9fb7c-414c-489d-898f-d2e0c53464c9”,
“name”: “Download Image2”,
“type”: “n8n-nodes-base.awsS3”,
“typeVersion”: 1,
“position”: [-560, 2400],
“credentials”: {
“aws”: {
“id”: “REDACTED_CRED_ID”,
“name”: “AWS account (sanitised)”
}
}
},
{
“parameters”: {
“conditions”: {
“options”: {
“caseSensitive”: true,
“leftValue”: “”,
“typeValidation”: “strict”,
“version”: 1
},
“conditions”: [
{
“id”: “8a2b9e5f-c3d4-4f5e-9876-1a2b3c4d5e6f”,
“leftValue”: “={{ $json.Key }}”,
“rightValue”: “\.(jpg|jpeg|png|gif|bmp|webp)$”,
“operator”: { “type”: “string”, “operation”: “regex” }
}
],
“combinator”: “and”
},
“options”: {}
},
“id”: “dd9801a9-59b0-4007-8a5b-6407173be7b2”,
“name”: “Filter Image Files2”,
“type”: “n8n-nodes-base.filter”,
“typeVersion”: 2,
“position”: [-340, 2400]
},
{
“parameters”: {
“batchSize”: “={{ $json.batchSize }}”,
“options”: { “reset”: false }
},
“type”: “n8n-nodes-base.splitInBatches”,
“typeVersion”: 3,
“position”: [-1000, 2220],
“id”: “70edb950-321f-4882-9641-f3cca8afa0b1”,
“name”: “Loop Over Items”
},
{
“parameters”: {
“jsCode”: “// — Function A: before loop —\nconst wf = $getWorkflowStaticData(‘global’);\nwf.allImages = items;\nwf.currentIndex = 0;\nwf.batchSize = Math.min(200, items.length);\nreturn items.map(i => {\n i.json.batchSize = wf.batchSize;\n return i;\n});”
},
“type”: “n8n-nodes-base.code”,
“typeVersion”: 2,
“position”: [-1240, 2220],
“id”: “ba6db312-6e0c-4337-b7ba-6e33b6e04b7e”,
“name”: “Code1”
},
{
“parameters”: {
“jsCode”: “// — Function B: inside loop —\nconst wf = $getWorkflowStaticData(‘global’);\nwf.processedBatches = (wf.processedBatches ?? 0) + 1;\nconst total = wf.allImages.length;\nconst size = wf.batchSize;\nconst expected = Math.ceil(total / size);\nconsole.log(Batch ${wf.processedBatches}/${expected} • items: ${items.length});\nreturn items;”
},
“type”: “n8n-nodes-base.code”,
“typeVersion”: 2,
“position”: [-120, 2400],
“id”: “06dcb0ec-b3c8-43f6-8fee-5896d422b0f2”,
“name”: “Code2”
},
{
“parameters”: { “amount”: 30 },
“type”: “n8n-nodes-base.wait”,
“typeVersion”: 1.1,
“position”: [-780, 2320],
“id”: “176c297e-c5a3-47db-a7a6-4683ce5effe1”,
“name”: “Wait”,
“webhookId”: “REDACTED_WAIT_ID”
}
],
“connections”: {
“List S3 Images2”: { “main”: [[{ “node”: “Code1”, “type”: “main”, “index”: 0 }]] },
“Download Image2”: { “main”: [[{ “node”: “Filter Image Files2”, “type”: “main”, “index”: 0 }]] },
“Filter Image Files2”: { “main”: [[{ “node”: “Code2”, “type”: “main”, “index”: 0 }]] },
“Loop Over Items”: { “main”: [, [{ “node”: “Wait”, “type”: “main”, “index”: 0 }]] },
“Code1”: { “main”: [[{ “node”: “Loop Over Items”, “type”: “main”, “index”: 0 }]] },
“Code2”: { “main”: [[{ “node”: “Loop Over Items”, “type”: “main”, “index”: 0 }]] },
“Wait”: { “main”: [[{ “node”: “Download Image2”, “type”: “main”, “index”: 0 }]] }
},
“pinData”: {},
“meta”: {
“templateCredsSetupCompleted”: true,
“instanceId”: “REDACTED_INSTANCE_ID”
}
}