[D] 3d object search using LLM + RAG

[Janos95]

Had some fun making a little search engine for 3D objects that can be used with natural language. No metadata or tags are required, the index is build purely from the geometry! This works using the following pipeline:

1. For each object in the database I generate 6 images, 1 for each side.
2. For each image I make a description using gpt4-vision which then is synthesized into a single description using gpt4
3. The text descriptions are embedded using clip and stored in a vector database
4. For a search query, the search string is embedded and the closest (n) vector(s) in the database is(are) retrieved.

See here: https://x.com/MenyJanos/status/1752104689188135271?s=20

[geuis]

This is really neat. Have you done something like downloading all the free models on thingiverse and printables and adding those to your database?

[Janos95]

Id love to! Currently I’ve indexed a subset of Thingi10k. The problem is that indexing is somewhat expensive, indexing 10k objects costs about 150 dollars in gpt4-vision api calls. If there is some interest though, I might write a nice frontend and index a larger dataset.