Scrape query from serper api than add airtable and ZEP

I first create the user chat entry as an airtable record.
I search this entry with Serper.dev. In the json result, I separate the organicsearch, relatedsearch and peoplealsoask directories with the split out option and handle them with different loops.

organicsearch
In this result, the sitelinks section is a separate directory and I check if it exists or not.
I exclude Youyube results.
I process the records in the Airtable sitelinks table.
I waited 4 seconds, then I processed the user input on the sitelink record. I did not use the patch method to pull this page content here. I added it to the ZEP vector database I had previously set up, using the user input metadata. The interesting thing here is that when I asked open AI to retrieve the company name from an open AI record, I got the response that it was unethical, so I couldn’t do it. The sitelinks directory comes up empty in basic searches.

People also ask
What people searched for in this search and what results they got
scrnli_23-02-2024 19-16-38

Again, I add the record to a separate airtable table, wait a while and process the user input. I use metadata when adding to the ZEP vector database.
scrnli_23-02-2024 19-22-01

relatedSearch
Thinking that these results will be useful to me, I add them to the airtable and the ZEP vector database.

Since Open AI would be an expensive solution for embedding in the ZEP vector database, I chose the HuggingFace BAAI/bge-small-en-v1.5 model. This model uses 384 embedding dimensions.

With the Default Data Loader option, metadata entry is very simple with no-code n8. By adding Token Splitter it works fine with the following settings.

I installed n8n and ZEP on docker with Oracle cloud free tier. Huggingface and openAI token required.

1 Like

Hey @Eray_Bartan,

Welcome to the community :raised_hands:

Do you have a workflow to share with everyone? It sounds very interesting.

1 Like

Hi John i didnt share my workflow cause self hosted free version.

2 Likes

Thanks for sharing this, Eray! If you’re comfortable giving this away, you could share the workflow on the Creator Hub as well: n8n workflow templates

You can sign up here to get started.

Right it would be great!

1 Like

i did create one and waiting for review approve.

Thanks for interest.

1 Like