Need to update a variable with a field from a http request node, but only the last one

Questions
1

Okay, so, I am using a http request node to pull historical data from a free government api site. They limit the data to 100 records, even if you sign up and get a key. But they will let you do refined pulls based on date.
So the date looks something like this
Date(YYYY-MM-DD), SomeNumber
Now, I start out using a set node and set last_date to todays date and then issue the http request as http://www.somesite.gov/folder/data?$where=date<{{last_date}} and get the first 1000 results.
Now, how do I update the last_date variable from the initial set node to be the last date returned from the data set so I can loop back through after I write the current data out?

Describe the problem/error/question

What is the error message (if any)?

Please share your workflow

(Select the nodes on your canvas and use the keyboard shortcuts CMD+C/CTRL+C and CMD+V/CTRL+V to copy and paste the workflow.)

Share the output returned by the last node

Information on your n8n setup

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:
2

Hy @Greg_Golden !

Welcome to n8n community!

One approach would be :

​- Start / Initial Set: Sets the starting last_date (e.g., today).

​- HTTP Request: Pulls the first 100 records based on the initial last_date.

​- Data Processing/Write Node: Writes the 100 records out (e.g., Google Sheets, File, etc.).

​- Updating Set: Extracts the date from the last record of the HTTP Request output and sets the new last_date.

​- If Node (Loop Condition): Checks if the HTTP Request returned 100 records (i.e., {{$json.length === 100}}).

​*True Branch: Connects back to the HTTP Request Node (Step 2) to continue the loop.

*​False Branch: Connects to the End of the Workflow (loop terminates).

If you share your Set and HTTP node here with some pinned data with the output, we can take a look better.

Cheers!