Need help to store data in vector db (supabase)

Questions
, , , ,
1

Hello I am building the workflow to fetch the wordpress site content and it is going to store in vector db so later on I can use that data to feed AI agent.

Describe the problem/error/question

I chunk the data and it is ready for embedding with the help of Hugging Face. Once data is emebedding It return the data in array but I need to append the reponse with the existing metadata object.

What is the error message (if any)?

Please share your workflow

Data Chunk Node:

{
  "nodes": [
    {
      "parameters": {
        "jsCode": "function chunkText(text, size = 1200, overlap = 150) {\n  const chunks = [];\n  let start = 0;\n\n  while (start < text.length) {\n    const end = Math.min(start + size, text.length);\n    const chunk = text.slice(start, end).trim();\n\n    if (chunk.length > 0) {\n      chunks.push(chunk);\n    }\n\n    start += size - overlap;\n  }\n\n  return chunks;\n}\n\n\nconst items = $input.all();\n\nlet output = [];\n\nfor (const item of items) {\n\n  const input = item.json;\n\n  const text = input.clean_text || '';\n\n  if (!text) {\n    continue;\n  }\n\n  const chunks = chunkText(text);\n\n  chunks.forEach((chunk, index) => {\n\n    output.push({\n      json: {\n        id: input.id,\n        title: input.title,\n        slug: input.slug,\n        link: input.link,\n        modified: input.modified || '',\n        source_type: input.source_type || 'page',\n\n        chunk_index: index,\n\n        chunk_text: chunk,\n\n        // Later we will use this for upsert/update\n        chunk_hash: `${input.id}-${index}-${chunk.length}`\n      }\n    });\n\n  });\n}\n\nreturn output;"
      },
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        832,
        0
      ],
      "id": "2984be2c-5f97-4833-8d5e-a0912be70559",
      "name": "ChunkText"
    }
  ],
  "connections": {
    "ChunkText": {
      "main": [
        []
      ]
    }
  },
  "pinData": {},
  "meta": {
    "templateCredsSetupCompleted": true,
    "instanceId": "a12d2939883b6ee4645287d26116bd7830ea910b3e2205baee32d9bb5d782014"
  }
}


Hugging Face Node:


{
  "nodes": [
    {
      "parameters": {
        "method": "POST",
        "url": "https://router.huggingface.co/hf-inference/models/BAAI/bge-small-en-v1.5",
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [
            {
              "name": "Authorization",
              "value": "Bearer <<API TOKEN>>"
            },
            {
              "name": "Content-Type",
              "value": "application/json"
            }
          ]
        },
        "sendBody": true,
        "specifyBody": "json",
        "jsonBody": "={{\n{\n  inputs: $json.chunk_text\n}\n}}",
        "options": {
          "response": {
            "response": {
              "responseFormat": "json"
            }
          }
        }
      },
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.4,
      "position": [
        992,
        -144
      ],
      "id": "2f3d7b70-454c-4e1d-8345-33ef8e3b047c",
      "name": "Hugging Face API"
    }
  ],
  "connections": {
    "Hugging Face API": {
      "main": [
        []
      ]
    }
  },
  "pinData": {},
  "meta": {
    "templateCredsSetupCompleted": true,
    "instanceId": "a12d2939883b6ee4645287d26116bd7830ea910b3e2205baee32d9bb5d782014"
  }
}

Share the output returned by the last node

Response from Hugging Face Models:
[
-0.03779905289411545,
-0.027235977351665497,
0.04070237651467323,
0.02413654699921608,
-0.04879286512732506,
-0.016459722071886063,
-0.0015258010243996978,
0.07087898999452591,
-0.03721754252910614,
0.019904209300875664,
0.022374162450432777,
-0.05740947648882866,
-0.017154762521386147,
0.04529924318194389,
0.0001327882637269795,
-0.023566097021102905,
0.008483237586915493,
0.013999130576848984,
-0.057878654450178146,
0.0066631208173930645,
0.034953076392412186,
-0.02018037624657154,
-0.013172738254070282,
-0.030672265216708183,
0.007248119916766882
]

Vector db schema:
create table public.site_chunks (
id bigserial not null,
url text not null,
title text null,
chunk_text text not null,
chunk_hash text not null,
source_type text not null,
modified_at timestamp with time zone null,
created_at timestamp with time zone not null default now(),
updated_at timestamp with time zone not null default now(),
embedding public.vector null,
constraint site_chunks_pkey primary key (id)
) TABLESPACE pg_default;

create unique INDEX IF not exists site_chunks_url_hash_idx on public.site_chunks using btree (url, chunk_hash) TABLESPACE pg_default;

create index IF not exists site_chunks_source_type_idx on public.site_chunks using btree (source_type) TABLESPACE pg_default;

create index IF not exists site_chunks_modified_at_idx on public.site_chunks using btree (modified_at) TABLESPACE pg_default;

Note: As I provided single array output but I have 53 chunks text and I am going to embedding and API return the total 20352 items so I am not sure how to map all those data with my existing records.

Sample Input:
[
{
“id”: 1383,
“title”: “blog”,
“url”: “TEST”,
“content”: “”,
“modified”: “2026-07-29T23:06:08”,
“source_type”: “page”,
“clean_text”: “”
},
{
“id”: 1222,
“title”: “Laser Dentistry”,
“slug”: “”,
“link”: “”,
“date”: “”,
“clean_text”: “Contrary to popular belief, Lorem Ipsum is not simply random text. It has roots in a piece of classical Latin literature from 45 BC, making it over 2000 years old. Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia, looked up one of the more obscure Latin words, consectetur, from a Lorem Ipsum passage, and going through the cites of the word in classical literature, discovered the undoubtable source. Lorem Ipsum comes from sections 1.10.32 and 1.10.33 of “de Finibus Bonorum et Malorum” (The Extremes of Good and Evil) by Cicero, written in 45 BC. This book is a treatise on the theory of ethics, very popular during the Renaissance. The first line of Lorem Ipsum”
} ]

Note: I am sending clean_text in the body parameter for embedding.

Expected output:
{ "url": "", "title": "Laser Dentistry", "chunk_text": "LASER DENTISTRY...", "chunk_hash": "1222-0-1200", "source_type": "page", "modified_at": null, "embedding": "[-0.037799,-0.027235,0.040702,...]" }

Information on your n8n setup

  • n8n version:
  • Database (default: SQLite):
  • n8n EXECUTIONS_PROCESS setting (default: own, main):
  • Running n8n via (Docker, npm, n8n cloud, desktop app): Azure App Service
  • Operating system: Linux
3

Hammad’s diagnosis is right — here’s the actual fix.

Set the HTTP Request node to return text instead of JSON:
Options > Response > Response Format = Text

That stops n8n splitting the array into 384 separate items. Then
add a Code node after it:

return $input.all().map(item => {
const src = $(‘ChunkText’).item.json;
return {
json: {
url: src.link,
title: src.title,
chunk_text: src.chunk_text,
chunk_hash: src.chunk_hash,
source_type: src.source_type,
modified_at: src.modified || null,
embedding: JSON.parse(item.json.data)
}
};
});

$(‘ChunkText’).item uses paired item tracking, so each response
stays linked to the chunk that produced it. This only works if the
array isn’t split, which is why the Text format matters.

Two other things worth fixing now rather than later.

Your column is declared as vector with no dimension:

embedding public.vector null

That works for inserts but you can’t build an index on it.
bge-small-en-v1.5 outputs 384 dimensions, so:

embedding vector(384)

Without a dimension you can’t create HNSW or IVFFlat, and without
an index every similarity search scans the whole table. Fine at 53
chunks, not at 50,000.

Second, your chunk_hash includes chunk length:

${input.id}-${index}-${chunk.length}

You noted this is for upsert later. It won’t work. If a page is
edited and a chunk’s length changes, the hash changes, so your
unique index on (url, chunk_hash) won’t match and you get a new row
instead of an update. The old chunk stays. A few months in, you
have both versions of every edited page in the table and the agent
retrieves from both.

Use a stable identifier for the row:

chunk_hash: ${input.id}-${index}

And add a separate content_hash column with a SHA of the chunk text
if you want to detect actual content changes.

4

Hi @Akash22
In the Hugging Face node turn on Options > Response > Include Response Headers and Status. The vector then arrives nested under body, one item per chunk, and stays intact.
Then add a Merge node with Mode set to Combine and Combine By set to Position, ChunkText into input 1 and Hugging Face into input 2. Both sides produce 53 items in the same order, so each output row carries its own metadata and its own vector, ready to map straight into the Supabase node with no Code step.
Send the embedding column as a string:

{{ JSON.stringify($json.body) }}

PostgREST turns a JSON array into Postgres array syntax, which the vector type rejects. Supabase’s own generated types expose a vector column as a string for the same reason.

More on the retrieval side once the rows are in:
https://axshul.site/n8n/guide/vector-stores-and-rag/

6

Hi Akash,

Hammad has your actual question in post 2 and the 53 by 384 arithmetic is right.

Two other things in what you posted are worth more of your attention than the mapping problem, because they will not announce themselves later.

Your chunk_hash is id-index-length, and your unique index is on (url, chunk_hash) with a comment saying you will use it for upsert. That hash contains no content. If someone edits a page and the chunk comes out the same length, a price going from 50 to 60, a swapped word, a typo fixed with a word of equal length, the hash is identical. The upsert matches, nothing updates, and the vector store keeps serving the old text. Nothing errors and the workflow stays green. Given one of your sample titles is Laser Dentistry, the stale thing could easily be a price or a policy.

The other direction is not much better. When the length does change, the hash changes, so it inserts a new row instead of updating the old one. The previous chunk is never removed, and after a few re-ingests you have several versions of the same passage competing in retrieval. The agent can then quote text that is no longer on the site at all.

Both are fixed by the same two changes. Hash the content itself rather than its length, sha256 of chunk_text is plenty. Then on each re-ingest, delete rows for that url whose hash is not in the set you just produced. That makes the index a mirror of the site rather than an accumulation of everything it has ever been.

One smaller thing while you are in the schema. Your embedding column is declared as vector with no dimension. pgvector will not build an ivfflat or hnsw index on that, so every query is a sequential scan over the whole table. At 53 chunks you will not notice. At 50,000 you will. Since bge-small is 384, vector(384) is what you want.

Adam

7

Thank for providing detailed information and good catch on chunk_hash key and vector size. I corrected my workflow and I keep my response in JSON format but I enabled the Include Response Headers and Status on HTTP node.

8

Thank you for providing detailed information and the way you mentioned I did it and it works fine.

9

Thank you so much for information. I will check and fix the workflow accordingly.