How to handle large volume of data in n8n

  1. Guys, I’m facing a challenge here with the n8n. I need to make a request to an API and receive from it the property data available in the system (it’s a real estate system) and with this data, I want to populate a knowledge base in dify, to be consulted by an agent. However, the system’s API limits me to pulling only 20 properties with each request, and they come in a single event and in a very large Json size. Does anyone know how I can process this data to populate the base in dify?

This is the json I receive as a response to each request

[
  {
    "json": {
      "quantidade": 769,
      "menorvalor": 0,
      "maiorvalor": 0,
      "menorarea": 0,
      "maiorarea": 0,
      "lista": [
        {
          "codigo": 8623,
          "codigomae": 0,
          "empreendimento": false,
          "empreendimentofilho": false,
          "nomeempreendimento": "",
          "siteempreendimento": "",
          "codigocondominio": 0,
          "nomecondominio": "",
          "titulo": "Casa à venda em Araranguá no bairro Coloninha",
          "metadescription": "Casa à venda em Araranguá no bairro Coloninha, com 4 quartos, 1 suíte, 3 banheiros, 2 vagas, por apenas 550.000,00. Confira mais opções. Os melhores imóveis você encontra aqui.",
          "codigoauxiliar": "",
          "unidade": 5766,
          "nomeunidade": "PROJETAR IMOVEIS ARARANGUA",
          "telefoneunidade": "(48) 98500-7999",
          "emaileunidade": "[email protected]",
          "finalidade": "Venda",
          "codigodestinacao": 1,
          "destinacao": "Residencial",
          "codigotipo": 1,
          "tipo": "Casa",
          "codigotipo2": null,
          "tipo2": "",
          "situacao": "Vago/Disponível",
          "valor": "R$ 550.000,00",
          "valoranterior": "R$ 0,00",
          "valorm2": "R$ 1.122,45",
          "valorminimo": "R$ 550.000,00",
          "valormaximo": "R$ 550.000,00",
          "taxaadm": "0,00",
          "taxaintermediacao": "0,00",
          "taxacomissao": "6,00",
          "rentabilidade": "0,00",
          "valorcondominio": "R$ 0,00",
          "valormaiscondominio": "R$ 550.000,00",
          "valormaiscondominiomaisiptu": "R$ 550.000,00",
          "indiceiptu": "",
          "valoriptu": "R$ 0,00",
          "valoriptuanual": "R$ 0,00",
          "valorspu": "R$ 0,00"
        }
      ]
    }
  }
]

and much much more …

It looks like your topic is missing some important information. Could you provide the following if applicable.

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:

Do you need all the data from the json? If not, use the Set Node to reduce the data, than split node.