Memory in looping scenario

[ezraluandre]

Hi, I have a question regarding how agent behave in looping scenario. The context is that I have a huge amount of data and since ai model can handle only for a certain context then I clustered the content in the form of around 10k tokens.

Since the agent task is to filter the data, then for each looping scenario it needs to understand the data that has already been filtered previously from the start of it. Here comes the usage of memory, but I’m not sure how the memory node behave when agent is start to run. My question is,

1. when agent start to run it always read the chat memory from previous chat (depending on the context window length), means I don’t need to do anything to give agent previous processed data or,
2. I need to pass the previous chat directly to agent,
   which one of this scenario that is correct?

[Parintele_Damaskin]

@ezraluandre

when agent start to run it always read the chat memory from previous chat (depending on the context window length), means I don’t need to do anything to give agent previous processed data or,

You nailed it !

If you think that data is too large to be “remembered” and accurate conversation , create summary and use it for new conversations.

What is your case ? 10k tokens per turn?

[ezraluandre]

Thanks @Parintele_Damaskin for your help to clarify it, the problem is the data can’t be summarize since it is a deduplication agent means I need previous data to understand if there’s the same data previously.

The total data I need to process is around 400+ and I clustered it based on 10k tokens and it still producing 6 data cluster.