I am trying to handle Dropbox “pagination” using cursors.
I have a problem where the items are multiplied in the output.
I have seen solutions for pagination, but they only handle one item in the input - in this case one folder path. However, I want to handle multiple folder paths with separate cursors and combine all the entries from the response together at the end.
Unfortunately, in the last code node where I’m merging the data together, I get all the items multiplied by the number of input paths.
I have included an example workflow. The important thing is to provide an existing path on Dropbox that contains enough files/folders to enter the loop at least once.
And by the way, I tried to reproduce this situation on mocked-up data, and it worked as expected. I have no idea where the issue is. Thanks for the help in advance.
Our Dropbox node can return all, and handle recursive folder listings - is there any particular reason why you’re using the HTTP Request node here? If you use the Dropbox node instead, does this fix this up?
You could also try something like this in your code node:
let results = [],
i = 0;
do {
try {
results = results.concat($("Has More").all(0, i));
} catch (error) {
return results;
}
i++;
} while (true);
The reason why I’m using the HTTP Request node is that I store a cursor value and use it to detect changes in folders and subfolders. The mechanism is described here in the Dropbox docs.
I cannot see the option to use a cursor in the Dropbox node. In the example workflow, the part of storing the cursor is skipped for debugging, and information about all files and folders is always fetched. However, in my actual workflow, I only fetch all changes once and then I can fetch only the changes.
Your code changed the behavior, but the results seem to be strange.
I have a result that is close to the expected one. I attached Item Lists to both outputs of the Has More If node, and then in the Code node, I used your code. In the last run, I can see the expected number of items (339 from list_folder + 1784 from list_folder/continue = 2123), but the runs are still not merged together.
Another strange thing in this case is that depending on where the Item Lists node is (below or above the Has More node), the result is different: 10833 vs 10538.
I wouldn’t put bth the true and false branch into the item lists node as it will trigger it for both sides so you will have 9 exucutions on that node that it might not need which is going to trigger the code node 9 times which is going to do all sorts of crazy things.
It looks the loop itself is going to be the problem so normally I would have the item lists node after the HTTP request node before the If node then I would use the Code at the end to do the merging on the Item Lists node from the false branch. Have you tried it this way round instead yet?
I haven’t tried this approach because in the IF node, I need to access the has_more parameter, which can come from both HTTP nodes. If they are connected directly to the IF node, I simply use {{ $json.has_more }}. However, if there is a List Items node just before the IF node, I don’t know how to access has_more from either the first HTTP node or the second one. Is there any way to do this?
Another thing is that when I put the Split Item node before the IF node, the number of output items from the IF node will determine the input to the list_folder/continue node. This means that the number of requests to Dropbox will be huge, as it will be equal to the number of entries (number of files) from Dropbox.
I also tried using a NoOp node between the HTTP node and the IF node, but the result is the same. I believe the problem is that the code does not actually merge the runs. From what I can see, there are two runs in the output, each with 7 items. As far as I understand, there should only be one run with all the items together.
It would be once for each input, because you have the 2 inputs it gets trickier and you need to redesign your loop to allow for this.
There is probably a better way to build the loop but I would need to find some time to get a solution working unless you can provide the output data from the nodes.
If you use the Dropbox node instead, does this fix this up?
