Support for Python/S3 integration

[Edoardo_Cantagallo]

Hi everybody! I am a new n8n user and so far I am having a really promising experience. I have a question regarding one task I need to automate.

I have a Python scraper and a S3 bucket where I need to store the data. For each record that I get in the scraper (via requests library) I need to check if that specific id is in S3. If it is just skip the record, otherwise create the folder and save the records in it. How can I solve this task?

thank you for your help!

[n8n]

It looks like your topic is missing some important information. Could you provide the following if applicable.

• n8n version:
• Database (default: SQLite):
• n8n EXECUTIONS_PROCESS setting (default: own, main):
• Running n8n via (Docker, npm, n8n cloud, desktop app):
• Operating system:

[ria]

Hi @Edoardo_Cantagallo

Thanks for posting here and welcome to the community!

I think you can achieve this workflow by using a couple of different nodes:

• HTTP Request Node to fetch your scraper results
• SplitInBatches (Loop) Node for handling each record individually
• S3 Node check if the specific ID exists
• IF Node handle the two scenarios (ID exist vs. ID doesn’t exist)
• S3 Node (from “No” in IF Node) - upload the records

Give it a try and let us know how you get on or if you encounter any specific blocking points!

[Edoardo_Cantagallo]

Hi Ria! Thank you for your support, my Python script iterates over multiple dates and I would need to repeat this operation for each date.

For example, the script for 1st of January returns some records, then 2nd of january ecc. ecc. How Can I achieve this extra step?

[ria]

This should be easy enough if you hook up prior to your Python script a code node that will generate an array of date strings and then use another SplitInBatches (Loop) node to process each date one by one. You can then pass the current date as an argument or query parameter for your Python scraper.

Hope this helps!