Is it possible to process a large number of dynamoDB records in batches

I am running a dynamodb query which returns a few thousand results. In my browser I get out of memory errors when I select “get all”. I tried to reduce it to only return 50 but I cannot determine how to paginate. There should be an output paramter “LastEvaluatedKey” which can then be sent as an input parameter “ExclusiveStartKey” so I figured I could use an IF block and see if lastevaluatedkey existed, run the query again and send exclusivestartkey but I cannot find lastevaluatedkey in the output even when reducing the number of records returned.

(Side question: When I click execute workflow - is it running fully on the server? Does it matter that my browser ran out of memory? The process seems to be getting stuck somewhere, is it possible to view output logs of an executing workflow if the browser loses connection?)

Hi @napter, I am not familiar with DynamoDB but the node does have a Simplify option and I suspect this could be what removes the output value you expect. Perhaps you can try disabling it?

image

When I click execute workflow - is it running fully on the server? Does it matter that my browser ran out of memory? The process seems to be getting stuck somewhere, is it possible to view output logs of an executing workflow if the browser loses connection?

Yes, it would run on the server, but the data you’re processing will be sent data between server and browser. For large datasets it’s important to know n8n doesn’t monitor it’s own memory consumption though. Meaning large amounts of data can crash the process executing your workflow even when executed fully on the server (if the process runs out of available memory). So you might want to take a look at the server log output (and possible also enable debug logging).

I’m afraid I was not clear. I am not using the simplify option (because it is a little broken and removes properties using the Map data type).

I am able to get all the results. However, it is a large dataset and tends to run out of memory on the browser. What I want to do is paginate the results - which will resolve that problem. DynamoDB supports pagination but it seems n8n is not exposing the fields returned by dynamodb that are necessary for pagination.

Thanks @napter, I am afraid I don’t have a great suggestion on how to achieve this, but perhaps someone with more experience might be able to chip in :frowning:

So for now I’ll convert your question into a feature request, so you and other users can vote on having the LastEvaluatedKey value exposed by n8n’s DynamoDB node.

Can you point me at the code where the dynamodb integration is implemented? Perhaps I can fix it.

1 Like

Sure thing, pull requests are greatly appreciated! We also have a hackathon coming up, in case you want to publish your changes as a new community node: 🏆 Announcing our first n8n community node-athon!

The existing DynamoDB node lives under n8n/packages/nodes-base/nodes/Aws/DynamoDB at master · n8n-io/n8n (github.com)

1 Like