Generate image embedding and store in vectordb

[davide445]

Describe the problem/error/question

I have seen this question asked only once in this post, but due my setup it’s not constrained to AWS Bedrock I think worth posting a different one.

My goal is generate image embedding using multilangage models (the search will be performed in Italian) like OpenAI CLIP or MS Florence-2 for all images available in a directory on my PC, and store them in a vectorDB like Qdrant where I can later execute the semantic search based on a chat request.

Looking in Langchain there are a couple of models already integrated

OpenClip | 🦜️🔗 LangChain

DeepInfra | 🦜️🔗 LangChain

That provide this multilanguage model

sentence-transformers/clip-ViT-B-32-multilingual-v1 - Demo - DeepInfra

Wanted to kindly ask if the path obtaining this is still using the custom code node, or there is some out of the box option.

What is the error message (if any)?

Please share your workflow

(Select the nodes on your canvas and use the keyboard shortcuts CMD+C/CTRL+C and CMD+V/CTRL+V to copy and paste the workflow.)

Share the output returned by the last node

Information on your n8n setup

• n8n version: 1.51.1
• Database (default: SQLite): Qdrant
• n8n EXECUTIONS_PROCESS setting (default: own, main):
• Running n8n via (Docker, npm, n8n cloud, desktop app): npm
• Operating system: KDE Neon 6.0 (Ubuntu 22.04)

[Jim_Le]

Hey @davide445 welcome!

A bit of Yes and No.

• Yes - there is nothing built-in for DeepInfra yet so you will need to use custom code to connect with the service.
• No - unfortunately, even if you could use the Langchain DeepInfra modules using custom code, you still wouldn’t be able to use it with images. To supply an image to DeepInfra, you’d need to call the API directly. See docs.

Alternatively…

• @oleg did a fantastic demo for multimodal embeddings in the recent Local AI hangout. His approach uses GPT-4o to generate semantic keywords for the image he uploads which are then used for the embeddings. His workflow used only built-in nodes so no custom code is necessary.
• I did an alternative implementation of image search using an object detection model. Not as cool as using a vector store of course, but could be useful for some use-cases.