i would recommend not using Vector Store. As for the what to use instead - that depends on the amount information you need to pass to the AI for generating the text. If it is relatively small - just combine all the pieces and send it in the request, but if that is going to exceed the limit of the input token budget - some of them support for 128000 tokens, some 200000, some are smaller and some are larger (like Gemini 1.5 Pro).
Your alternatives are training and using a custom model, if you need all that knowledge, or context streaming with summarization, but these are pretty advanced techniques. You will need to educate yourself on how to do that.
I’m not that advanced in n8n. I scrape tweets, which have around 1500-2000 words on average. I want to store them in a database. The Ai writes a new tweet based of the database. It should use the entire database.
I don’t want summaries of the tweets. I want completely new tweets based of the tweets that I fetched. I need to do this because I want the Ai agent to be extremely specific with the topics and chosen words
1500-2000 words will yield 2000-3000 tokens, which (even with an additional context in the form of a system prompt to instruct how to generate the output) is significantly lower than what fits in the context window, so you should be fine with just combining all the tweets and sending them directly to the model.
Ah, I see. 45000-60000 tokens (100+ pages of single spaced text) of input can be more challenging, but still should fit into the context window of most modern models. I’ve never worked with inputs this big before, but I’d say you could easily give it a try and see what comes out.
Optimizing for cost effectiveness needs to come from a specific need. 60-80k input tokens request will cost you between 20 and 80 cents, depending on the model. IS that much? I don’t know.
To answer your question though, as long as you retrieve all this data and feed it to the model it is going to be pretty much the same cost, which ever way you do it. If you need to make it more cost effective, you can:
Ok I think it will not work as I thought. I was hoping that I can fetch all tweets, save them into supabase and ChatGPT writes a tweet with the supabase knowledge. Or can I do that? Without telling chatgpt choose that exact tweet
I sort of already answered that, both how it would look like and how much it would cost. Not even sure why you need the database here - you scrape, combine all 30-50 tweets and send to the model as input. It will cost you money. If money is the issue - I also included what you may want to try to reduce the cost.
1 tweet has 3000 tokens, I fetch 30 at least per excecution and save them into my database