Hi. I have been trying to get batch information from a database for days now and really struggling to end up with a single list of the outputs which I then need to process onwards. I have got to the point of storing them in a PostgreSQL table but now the workflow gets about a third of the way through and then just stops (completes) the workflow. There are over 3300 records in total. Can anyone tell me where I am going wrong here? Thanks!
Hey there! Could you please send use the json of the workflow and the version of your n8n?
For me whenever this happens its usually because some data didn’t come through in the run (It’s important to check via the executions tab where it failed and what the data was).
Here is how I currently troubleshoot:
- Executions tab - Was the data not found or badly formatted?
- In several cases it was just that the data was EMPTY and so I had to change my node to ALWAYS do an output (If you go inside the various nodes, go into the settings and make sure its always generating an output even if its blank - once again this depends on the flow and what you want. In some cases you actually want it to fail on an empty to CATCH which data is giving you a null value)
Once I find the data that is causing the issue I either edit the data OR I edit the flow to accomodate the data.
Check and see 
Hi. Version is 1.80.3 self-hosted. JSON below:
Have just noticed that there looks to be a lost connection with the PostgreSQL server (see top right). Have now observed this twice at about the same time so suspect this is the issue?
It could be a timeout issue potentially (Once again, if you don’t notice it failing on data). In that case what I’ve been learning is that having a WAIT node may do the trick.
If anyone else has an idea please chime in as well, I’m just thinking that could be the issue if its a connection problem (Using a WAIT node to give it time to load from the database).
Also make sure you check the executions list to see where it failed imo. That usually gives an idea.
Hey, I’m having trouble trying to copy paste the workflow. Could you please use the triple backticks (```) when adding your workflow? That way i gets automatically displayed. Thanks
"name": "Store all BMS Service Items",
"nodes": [
{
"parameters": {},
"type": "n8n-nodes-base.manualTrigger",
"typeVersion": 1,
"position": [
140,
0
],
"id": "1169d60a-2c5c-4fbd-a999-294c8a34e04f",
"name": "When clicking ‘Test workflow’"
},
{
"parameters": {
"method": "POST",
"url": "https://fmaws.vitalcertificates.co.uk/fmi/data/v1/databases/VC Branch Management/sessions",
"authentication": "genericCredentialType",
"genericAuthType": "httpBasicAuth",
"sendHeaders": true,
"headerParameters": {
"parameters": [
{
"name": "Content-Type",
"value": "application/json"
}
]
},
"options": {}
},
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [
380,
0
],
"id": "8cf8a0ec-1856-469a-b565-03b29189af19",
"name": "HTTP Request",
"credentials": {
"httpHeaderAuth": {
"id": "SO772OpPzuKqORgc",
"name": "Header Auth Filemaker"
},
"httpCustomAuth": {
"id": "RrxkUhkNKPCb6JyJ",
"name": "Filemaker"
},
"httpBasicAuth": {
"id": "dsJ5HTRAU8uMbTB3",
"name": "Filemaker"
}
}
},
{
"parameters": {
"method": "POST",
"url": "https://fmaws.vitalcertificates.co.uk/fmi/data/v1/databases/VC Branch Management/layouts/Services_Detail/_find",
"sendHeaders": true,
"headerParameters": {
"parameters": [
{
"name": "Content-Type",
"value": "application/json"
},
{
"name": "Authorization",
"value": "=Bearer {{ $('HTTP Request').item.json.response.token }}"
}
]
},
"sendBody": true,
"specifyBody": "json",
"jsonBody": "={{ $json}}",
"options": {}
},
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [
1000,
0
],
"id": "bf9a8d0c-1dcf-4832-a10d-8d07bab48ad5",
"name": "HTTP Request1"
},
{
"parameters": {
"modelId": {
"__rl": true,
"value": "gpt-4o",
"mode": "list",
"cachedResultName": "GPT-4O"
},
"messages": {
"values": [
{
"content": "=***IMPORTANT: Use only UK-English, NOT US-English.*** \n\nYour role is to create a summarised datasheet on an attestation service offered by the company. Each service can differ on the basis of what document type it is, which country it was issued in, and which country it needs to be used in. Some services are further broken down by speed and/or variation of the document type and/or processing method. \n\nYou will be provided with a set of fields contained within the company's internal database. From these fields you should summarise, losing none of the actual detail, the service code and title, what the service is, how it is processed, and a range of costs based on the number of documents the client wishes to process. If quantity prices are not provided, just output the single fixed cost price. We also offer special rates for UK-issued documents (only) to registered corporate clients but these are by request. Please bear in mind that these prices exclude delivery and tax where appropriate. \n\nHere are the fields and details: \n\nService Code: {{ $json.fieldData.ServiceCode }}\n\nDescription: {{ $json.fieldData.Description }}\n\nIssuing Country: {{ $json.fieldData['Issue Country'] }}\n\nDestination Country: {{ $json.fieldData['Destination Country'] }}\n\nDocument Type: {{ $json.fieldData['Document Type'] }}\n\nService Variation: {{ $json.fieldData.ServiceVariant }}\n\nClient Advice Notes: {{ $json.fieldData.CommonClientAdvice }}\n\nClient Documents Required (If any): {{ $json.fieldData.ClientDocsRequired }}\n\nSpecific Service Notes: {{ $json.fieldData.ServiceNotes }}\n\nExpected Processing Time (not including shipping to/from our offices): \n{{ $json.fieldData.UKTimeframe }}\n\nPrices in GBP:\n\n1 document: {{ $json.fieldData.ukSellPrice_1 }}\n\n2 documents: {{ $json.fieldData.ukSellPrice_2 }}\n\n3 Documents: {{ $json.fieldData.ukSellPrice_3 }}\n\n4 Documents: {{ $json.fieldData.ukSellPrice_4 }}\n\n5 or more documents: {{ $json.fieldData.ukSellPrice_5 }}\n\nFixed Price: {{ $json.fieldData.c_OwnSell }}\n"
}
]
},
"options": {
"temperature": 0.5,
"maxToolsIterations": 0
}
},
"type": "@n8n/n8n-nodes-langchain.openAi",
"typeVersion": 1.8,
"position": [
3140,
260
],
"id": "18b8fa49-37be-4300-9458-d0f70b887949",
"name": "OpenAI",
"credentials": {
"openAiApi": {
"id": "bEeQ4sZ6w94tqHCT",
"name": "OpenAi account"
}
}
},
{
"parameters": {
"method": "DELETE",
"url": "=https://fmaws.vitalcertificates.co.uk/fmi/data/v1/databases/VC Branch Management/sessions/{{ $('HTTP Request').item.json.response.token }}",
"authentication": "genericCredentialType",
"genericAuthType": "httpBasicAuth",
"sendHeaders": true,
"headerParameters": {
"parameters": [
{
"name": "Content-Type",
"value": "application/json"
}
]
},
"options": {}
},
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [
2140,
20
],
"id": "afd997e9-a3f3-4540-b3bb-01274b6bc618",
"name": "HTTP Request3",
"credentials": {
"httpBasicAuth": {
"id": "dsJ5HTRAU8uMbTB3",
"name": "Filemaker"
}
}
},
{
"parameters": {
"method": "PATCH",
"url": "https://fmaws.vitalcertificates.co.uk/fmi/data/v1/databases/VC Branch Management/globals",
"sendHeaders": true,
"headerParameters": {
"parameters": [
{
"name": "Content-Type",
"value": "application/json"
},
{
"name": "Authorization",
"value": "=Bearer {{ $json.response.token }}"
}
]
},
"sendBody": true,
"specifyBody": "json",
"jsonBody": "={\"globalFields\":{\"Services::zz_BranchID\": 1}}",
"options": {}
},
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [
600,
0
],
"id": "9286f0d7-374e-4420-b3ab-72867aeb4675",
"name": "HTTP Request4"
},
{
"parameters": {
"mode": "insert",
"pineconeIndex": {
"__rl": true,
"value": "vitalconsular",
"mode": "list",
"cachedResultName": "vitalconsular"
},
"options": {
"pineconeNamespace": "vital_bms_uk"
}
},
"type": "@n8n/n8n-nodes-langchain.vectorStorePinecone",
"typeVersion": 1,
"position": [
3780,
260
],
"id": "59c1347f-a1bc-400b-b1ce-67f7181afdf9",
"name": "Pinecone Vector Store",
"credentials": {
"pineconeApi": {
"id": "qKnHWvIBJLCYAuBt",
"name": "PineconeApi account"
}
}
},
{
"parameters": {
"options": {}
},
"type": "@n8n/n8n-nodes-langchain.embeddingsOpenAi",
"typeVersion": 1.2,
"position": [
3700,
540
],
"id": "95bb79b7-fca8-4295-b16a-0479ec89b799",
"name": "Embeddings OpenAI",
"credentials": {
"openAiApi": {
"id": "bEeQ4sZ6w94tqHCT",
"name": "OpenAi account"
}
}
},
{
"parameters": {
"jsonMode": "expressionData",
"jsonData": "=Service Code: {{ $('Split Out').item.json.fieldData.ServiceCode }}. This describes the {{ $('Split Out').item.json.fieldData.Description }} service.\n\n{{ $('OpenAI').item.json.message.content }}",
"options": {
"pointers": "={{ $json[\"text\"] }}_chunk_{{ $itemIndex }}",
"metadata": {
"metadataValues": [
{
"name": "title",
"value": "={{ $('Split Out').item.json.fieldData.Description }}"
},
{
"name": "id",
"value": "={{ $('Split Out').item.json.fieldData.__Pk_Service }}"
},
{
"name": "service_code",
"value": "={{ $('Split Out').item.json.fieldData.ServiceCode }}"
},
{
"name": "issue_country",
"value": "={{ $('Split Out').item.json.fieldData[\"Issue Country\"] }}"
},
{
"name": "destination_country",
"value": "={{ $('Split Out').item.json.fieldData[\"Destination Country\"] }}"
},
{
"name": "document_category",
"value": "={{ $('Split Out').item.json.fieldData[\"Document Type\"] }}"
}
]
}
}
},
"type": "@n8n/n8n-nodes-langchain.documentDefaultDataLoader",
"typeVersion": 1,
"position": [
3840,
540
],
"id": "6c3be04e-ab5f-4c0b-b1d3-e36cd275b7c4",
"name": "Default Data Loader"
},
{
"parameters": {
"chunkOverlap": 200,
"options": {}
},
"type": "@n8n/n8n-nodes-langchain.textSplitterRecursiveCharacterTextSplitter",
"typeVersion": 1,
"position": [
3840,
720
],
"id": "d5bbb5f4-9855-4610-85b9-15ae73801d4c",
"name": "Recursive Character Text Splitter"
},
{
"parameters": {
"fieldToSplitOut": "message.content",
"options": {}
},
"type": "n8n-nodes-base.splitOut",
"typeVersion": 1,
"position": [
3500,
260
],
"id": "5f3b5ec5-1bc5-4219-8621-23feca38b023",
"name": "Split Out1"
},
{
"parameters": {
"jsCode": "// Determine if this is the first iteration or part of a loop\n// Default to 100 for the first batch after HTTP Request1\nlet currentOffset = $json.offset || 100;\n\n// Build the FileMaker query with proper pagination\nconst queryBody = {\n query: [\n { \"service_PARTNERS::PartnerStatus\": \"Active\" }\n ],\n limit: 100,\n offset: currentOffset\n};\n\n// Log the current offset for debugging\nconsole.log(`[Code] Using offset: ${currentOffset}`);\n\n// Return the query for the HTTP Request2 node\nreturn {\n json: {\n body: queryBody, // Required for FileMaker Data API\n offset: currentOffset // Pass through for reference\n }\n};"
},
"type": "n8n-nodes-base.code",
"typeVersion": 2,
"position": [
1300,
0
],
"id": "3d60445e-4c13-42da-8c0f-96d306a3f069",
"name": "Code"
},
{
"parameters": {
"jsCode": "// Retrieve the current offset (default to 0)\nlet offset = $json.offset || 0;\n\n// Build the basic JSON body with your query and limit\nlet body = {\n query: [\n { \"service_PARTNERS::PartnerStatus\": \"Active\" }\n ],\n limit: 100\n}\n\nreturn { json: body };"
},
"type": "n8n-nodes-base.code",
"typeVersion": 2,
"position": [
820,
0
],
"id": "3053f89a-6f98-43f0-837a-c902ab34a522",
"name": "Code1"
},
{
"parameters": {
"conditions": {
"options": {
"caseSensitive": true,
"leftValue": "",
"typeValidation": "strict",
"version": 2
},
"conditions": [
{
"id": "0edbabce-ea99-4570-918c-728c2ef6e4fa",
"leftValue": "={{ $json.continueLoop }}",
"rightValue": "true",
"operator": {
"type": "boolean",
"operation": "true",
"singleValue": true
}
}
],
"combinator": "and"
},
"options": {}
},
"type": "n8n-nodes-base.if",
"typeVersion": 2.2,
"position": [
1900,
0
],
"id": "a5990fd6-63aa-469e-8763-f3cd13b69c17",
"name": "If"
},
{
"parameters": {
"method": "POST",
"url": "https://fmaws.vitalcertificates.co.uk/fmi/data/v1/databases/VC Branch Management/layouts/Services_Detail/_find",
"sendHeaders": true,
"headerParameters": {
"parameters": [
{
"name": "Content-Type",
"value": "application/json"
},
{
"name": "Authorization",
"value": "=Bearer {{ $('HTTP Request').item.json.response.token }}"
}
]
},
"sendBody": true,
"specifyBody": "json",
"jsonBody": "={{ $json.body }}",
"options": {}
},
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [
1480,
0
],
"id": "18985872-170c-45c9-913c-7158739b861b",
"name": "HTTP Request2"
},
{
"parameters": {
"jsCode": "// Get the response data from HTTP Request2\nconst response = $json.response;\nconst dataInfo = response.dataInfo || {};\nconst foundCount = dataInfo.foundCount || 0;\nconst returnedCount = dataInfo.returnedCount || 0;\n\n// Get the current offset directly from the Code node\n// This is the offset value that was used in the current request\nlet currentOffset;\ntry {\n currentOffset = $('Code').first().json.offset;\n} catch (error) {\n currentOffset = 100; // Default fallback\n console.log(`[Code2] Error accessing Code node: ${error.message}`);\n}\n\n// Calculate the next offset for the following iteration\nconst nextOffset = currentOffset + returnedCount;\n\n// Determine if we should continue the loop\nconst continueLoop = nextOffset < foundCount;\n\n// Log information for debugging\nconsole.log(`[Code2] Current offset: ${currentOffset}, Records returned: ${returnedCount}`);\nconsole.log(`[Code2] Next offset: ${nextOffset}, Total records: ${foundCount}`);\nconsole.log(`[Code2] Continue loop: ${continueLoop}`);\n\n// Return data for the IF node\nreturn {\n json: {\n continueLoop: continueLoop, // Boolean for the IF node to evaluate\n offset: nextOffset, // For the next iteration if loop continues\n \n // Additional information for reference/debugging\n foundCount: foundCount,\n returnedCount: returnedCount\n }\n};"
},
"type": "n8n-nodes-base.code",
"typeVersion": 2,
"position": [
1740,
0
],
"id": "35b1ea5f-b22e-472f-a61d-8d8bd9487ceb",
"name": "Code2"
},
{
"parameters": {
"options": {}
},
"type": "n8n-nodes-base.splitInBatches",
"typeVersion": 3,
"position": [
2880,
260
],
"id": "34a7805a-f9b2-4b40-a6d7-253aa2936dde",
"name": "Loop Over Items"
},
{
"parameters": {},
"type": "n8n-nodes-base.merge",
"typeVersion": 3,
"position": [
1700,
260
],
"id": "b643271b-556d-4ef7-863d-784d869797c4",
"name": "Merge"
},
{
"parameters": {
"fieldToSplitOut": "response.data",
"options": {}
},
"type": "n8n-nodes-base.splitOut",
"typeVersion": 1,
"position": [
1900,
260
],
"id": "dfae9c84-287c-4ea7-84b2-7ea0b0ae6ce3",
"name": "Split Out2"
},
{
"parameters": {
"operation": "upsert",
"schema": {
"__rl": true,
"mode": "list",
"value": "public"
},
"table": {
"__rl": true,
"value": "data_collection",
"mode": "list",
"cachedResultName": "data_collection"
},
"columns": {
"mappingMode": "defineBelow",
"value": {
"id": "={{ $json.__Pk_Service }}",
"item_data": "={{ $json.item_data }}"
},
"matchingColumns": [
"id"
],
"schema": [
{
"id": "id",
"displayName": "id",
"required": false,
"defaultMatch": true,
"display": true,
"type": "number",
"canBeUsedToMatch": true
},
{
"id": "item_data",
"displayName": "item_data",
"required": false,
"defaultMatch": false,
"display": true,
"type": "object",
"canBeUsedToMatch": false
},
{
"id": "batch_id",
"displayName": "batch_id",
"required": false,
"defaultMatch": false,
"display": true,
"type": "string",
"canBeUsedToMatch": false
},
{
"id": "created_at",
"displayName": "created_at",
"required": false,
"defaultMatch": false,
"display": true,
"type": "dateTime",
"canBeUsedToMatch": false
}
],
"attemptToConvertTypes": false,
"convertFieldsToString": false
},
"options": {
"connectionTimeout": 300
}
},
"type": "n8n-nodes-base.postgres",
"typeVersion": 2.5,
"position": [
2420,
260
],
"id": "55cd32f0-b199-46bb-aad2-be03f96a75c9",
"name": "Postgres",
"credentials": {
"postgres": {
"id": "6pAW0I8XzLoTwtqS",
"name": "Postgres account"
}
}
},
{
"parameters": {
"operation": "select",
"schema": {
"__rl": true,
"mode": "list",
"value": "public"
},
"table": {
"__rl": true,
"value": "data_collection",
"mode": "list",
"cachedResultName": "data_collection"
},
"returnAll": true,
"options": {}
},
"type": "n8n-nodes-base.postgres",
"typeVersion": 2.5,
"position": [
2660,
-120
],
"id": "de818b9e-975f-4ca1-8fd9-8c63c102d273",
"name": "Postgres1",
"alwaysOutputData": false,
"credentials": {
"postgres": {
"id": "6pAW0I8XzLoTwtqS",
"name": "Postgres account"
}
}
},
{
"parameters": {
"mode": "runOnceForEachItem",
"jsCode": "const fieldData = $input.item.json.fieldData;\nconst jsonString = JSON.stringify(fieldData);\nconst base64String = Buffer.from(jsonString).toString('base64');\n\n// Extract the primary key field\nconst primaryKeyService = fieldData.__Pk_Service || null;\n\nreturn {\n json: {\n // For PostgreSQL JSON field, we need to create a proper JSON object\n item_data: {\n encoded_data: base64String,\n primary_key: primaryKeyService,\n encoding_type: \"base64\",\n content_type: \"application/json\",\n created_at: new Date().toISOString()\n },\n // Keep these separate for other uses if needed\n __Pk_Service: primaryKeyService,\n base64EncodedData: base64String\n }\n}"
},
"type": "n8n-nodes-base.code",
"typeVersion": 2,
"position": [
2160,
260
],
"id": "df29c1e0-dca2-426c-995a-a8fe036818ef",
"name": "Code4"
}
],
"pinData": {
"When clicking ‘Test workflow’": [
{
"json": {}
}
]
},
"connections": {
"When clicking ‘Test workflow’": {
"main": [
[
{
"node": "HTTP Request",
"type": "main",
"index": 0
}
]
]
},
"HTTP Request": {
"main": [
[
{
"node": "HTTP Request4",
"type": "main",
"index": 0
}
]
]
},
"HTTP Request1": {
"main": [
[
{
"node": "Code",
"type": "main",
"index": 0
},
{
"node": "Merge",
"type": "main",
"index": 1
}
]
]
},
"OpenAI": {
"main": [
[
{
"node": "Split Out1",
"type": "main",
"index": 0
}
]
]
},
"HTTP Request3": {
"main": [
[]
]
},
"HTTP Request4": {
"main": [
[
{
"node": "Code1",
"type": "main",
"index": 0
}
]
]
},
"Embeddings OpenAI": {
"ai_embedding": [
[
{
"node": "Pinecone Vector Store",
"type": "ai_embedding",
"index": 0
}
]
]
},
"Default Data Loader": {
"ai_document": [
[
{
"node": "Pinecone Vector Store",
"type": "ai_document",
"index": 0
}
]
]
},
"Recursive Character Text Splitter": {
"ai_textSplitter": [
[
{
"node": "Default Data Loader",
"type": "ai_textSplitter",
"index": 0
}
]
]
},
"Split Out1": {
"main": [
[
{
"node": "Pinecone Vector Store",
"type": "main",
"index": 0
}
]
]
},
"Code": {
"main": [
[
{
"node": "HTTP Request2",
"type": "main",
"index": 0
}
]
]
},
"Code1": {
"main": [
[
{
"node": "HTTP Request1",
"type": "main",
"index": 0
}
]
]
},
"If": {
"main": [
[
{
"node": "Code",
"type": "main",
"index": 0
}
],
[
{
"node": "HTTP Request3",
"type": "main",
"index": 0
}
]
]
},
"HTTP Request2": {
"main": [
[
{
"node": "Code2",
"type": "main",
"index": 0
},
{
"node": "Merge",
"type": "main",
"index": 0
}
]
]
},
"Code2": {
"main": [
[
{
"node": "If",
"type": "main",
"index": 0
}
]
]
},
"Pinecone Vector Store": {
"main": [
[
{
"node": "Loop Over Items",
"type": "main",
"index": 0
}
]
]
},
"Loop Over Items": {
"main": [
[],
[
{
"node": "OpenAI",
"type": "main",
"index": 0
}
]
]
},
"Merge": {
"main": [
[
{
"node": "Split Out2",
"type": "main",
"index": 0
}
]
]
},
"Split Out2": {
"main": [
[
{
"node": "Code4",
"type": "main",
"index": 0
}
]
]
},
"Code4": {
"main": [
[
{
"node": "Postgres",
"type": "main",
"index": 0
}
]
]
}
},
"active": false,
"settings": {
"executionOrder": "v1",
"callerPolicy": "workflowsFromSameOwner"
},
"versionId": "fb60dc1d-34ec-4185-b2af-4697d082aabb",
"meta": {
"templateCredsSetupCompleted": true,
"instanceId": "ae089b2b3e7b27ef840db210c133514d8bab95d260e915570f807f9e4e7bc7bf"
},
"id": "JlTztx5jb3CKKcxM",
"tags": []
}@SolomonChrist - yes, that sounded like a reasonable suggestion but didn’t change anything. Still got the connection lost message
To be clear, the PostgreSQL option is just one of several workarounds. I have to batch the data coming from the HTTP modules due to record limits, but then these batches are too large to feed into an OpenAI node later (I am trying to seed a RAG with our datastore). I tried the loop option to chunk these down but it only works for the first batch. This is incredibly frustrating
Hey there. I still cannot reproduce the issue since the workflow is not being copied properly. If I copy paste the workflow, it should look something like this:
Otherwise, I would recommend downloading it and pasting the resulting json here like so:
@danougill Is that better?
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.