How to handle large volume of data in n8n

  1. Guys, I’m facing a challenge here with the n8n. I need to make a request to an API and receive from it the property data available in the system (it’s a real estate system) and with this data, I want to populate a knowledge base in dify, to be consulted by an agent. However, the system’s API limits me to pulling only 20 properties with each request, and they come in a single event and in a very large Json size. Does anyone know how I can process this data to populate the base in dify?

This is the json I receive as a response to each request

[
  {
    "json": {
      "quantidade": 769,
      "menorvalor": 0,
      "maiorvalor": 0,
      "menorarea": 0,
      "maiorarea": 0,
      "lista": [
        {
          "codigo": 8623,
          "codigomae": 0,
          "empreendimento": false,
          "empreendimentofilho": false,
          "nomeempreendimento": "",
          "siteempreendimento": "",
          "codigocondominio": 0,
          "nomecondominio": "",
          "titulo": "Casa à venda em Araranguá no bairro Coloninha",
          "metadescription": "Casa à venda em Araranguá no bairro Coloninha, com 4 quartos, 1 suíte, 3 banheiros, 2 vagas, por apenas 550.000,00. Confira mais opções. Os melhores imóveis você encontra aqui.",
          "codigoauxiliar": "",
          "unidade": 5766,
          "nomeunidade": "PROJETAR IMOVEIS ARARANGUA",
          "telefoneunidade": "(48) 98500-7999",
          "emaileunidade": "[email protected]",
          "finalidade": "Venda",
          "codigodestinacao": 1,
          "destinacao": "Residencial",
          "codigotipo": 1,
          "tipo": "Casa",
          "codigotipo2": null,
          "tipo2": "",
          "situacao": "Vago/Disponível",
          "valor": "R$ 550.000,00",
          "valoranterior": "R$ 0,00",
          "valorm2": "R$ 1.122,45",
          "valorminimo": "R$ 550.000,00",
          "valormaximo": "R$ 550.000,00",
          "taxaadm": "0,00",
          "taxaintermediacao": "0,00",
          "taxacomissao": "6,00",
          "rentabilidade": "0,00",
          "valorcondominio": "R$ 0,00",
          "valormaiscondominio": "R$ 550.000,00",
          "valormaiscondominiomaisiptu": "R$ 550.000,00",
          "indiceiptu": "",
          "valoriptu": "R$ 0,00",
          "valoriptuanual": "R$ 0,00",
          "valorspu": "R$ 0,00"
        }
      ]
    }
  }
]

and much much more …

It looks like your topic is missing some important information. Could you provide the following if applicable.

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app):
  • Operating system:

Do you need all the data from the json? If not, use the Set Node to reduce the data, than split node.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.