I am trying to get all my products in shopify but if I am saying all products it keeps running no limit.
and I am puting 4000 limit it changes automatically to 250 that is an internal limit for shopify.
Therefore how can I get all the pages one by one ?
it didn’t work, i get the same error.
“Exceeded 2 calls per second for api client. Reduce request rates to resume uninterrupted service.”
Isn’t there a way to make the shopify node slow down the requests? i know there is a “wait” node but i don’t know if its relevant in this case? it seems to be useful to slow down the interactions between nodes, but not sure if it works to slow down a node itself …
Sadly no way to slow down the node itself, It looks like the requests per second are set based on shopify tier as well so if you had Shopify plus it is 4/second. It looks like maybe what we should do is look for the 429 header and wait briefly before sending the next request. For now it might be a case of using the HTTP Request node to manually call the API and loop through the results that way.
Add an HTTP Request, Set the credentials to use your Shopify ones, Set the URL and parameters to whatever Shopify expects and a set a limit of 250 (The max shopify allows). Next add a wait node and set it to 1 second then loop back to the HTTP Request node to grab the next 250.
You will need to add in some bits for the pagination as needed but the API docs for Shopify can be found here: Product
It looks like they will give you a link to use to get the next batch as well which is handy (Make paginated requests to the REST Admin API). I am just about to finish up for the day but I think this might be enough to get started with, If you don’t get anywhere let me know and I will see if I can free up some time to pop an example together.
You would be replacing the shopify node for now with the http request node, you may also need a set node to set the cursor value, an if node to check if there is more pages and a code node to merge all the items.
There are a few examples on here on pagination with APIs that could help.
Hello @Jon !
So i advanced, and now im able to do a HTTP request, get 50 products, dump them on angoogle sheet. On the other brand of the HTTP Request i get the link to do the HTTP GET request for the next 50 products , and do the same process to dum these 50 products ont he google sheet. However the issue i have no is that i don’t know how to iterate on the pagination without everytime having to add again the same nodes , again and again … do you have any insight?
That is where you would need the loop, Looking at the Shopify API documentation it will return the next URL to use so you would need to check if that exists and if it does run the HTTP Request node again with the new URL.
Hi @Jon , so as you said i looked at that thread and did the following loop:
The first HTTP request is to get it initiated, then the “IF” checks if there is a link pointing towards the next page with products ( so if there are more products to be requested). If it does then it gets the link of this next page and does an HTTP request with it to get the second page, then again, the IF checks if there is another page, and on one branch goes and dumps the products found on the google sheet, and on the other cleans the reponse header with the link to the next page, to use it for the next HTTP request.
There is probably a simpler way to do it however it works yey! the problem i have now is that i’m constantly running into google sheet api limit to do 50 requests … I already limited the shopify call to 50, to make sure that we have batches of 50 products to dump into google sheet, but after around 4 or 5 loops, it spits out the error again of "The service is receiving too many requests from you! Perhaps take a break"
To try to solve it i’m trying with a WAIT node of 30 seconds … but i wonder if there is a better way to do this ? isn’t 30 seconds too much ? i previously tried with 15 seconds and it still gave me that error.
30 seconds is a very long time but if Shopify is limiting your requests the only thing to do is wait as we can’t really do anything about that although you should be falling under their limits. In the workflow screenshot above did it throw the same error?
You are also coming off of the true branch of the if node twice is that intentional?
Hello! so upda:te it has been blocked after 14 iterations of the loop … so didnt finish exporting all my products from shopify
Regarding the If branch, yes it is intentional, as i use it to check if the link on the header resopnse contains “next” which would mean that tere is another page, and therefore i do the loop to clean the link and do another request …
i feel like there is probably a better way to do this but not sure
want’ed to update you, thanks for your help you helped me make it work. i’m sure it is not the most optimum wy but it is working here, feel free to lmk if there re any inmediate improvements we can do: