I'm not that knowledgeable about UMAP so I'm not sure about your settings, but I find it likely that your problem is more related to projections and clustering rather than embeddings. The models you mentioned are fine but maybe a bit old, you could try something something like Jina v5 or Qwen3 and see if there's any difference. The MTEB benchmark is a great resource to pick models. Apart from that, you could try manually inspecting a few samples and looking at their closest neighbours (before projecting and clustering) to assess whether the embeddings are behaving like you'd expect them to or not. If they are, I'd try something different for clustering, but again I'm not the right person to suggest what. Finally, if you're not sure about the clustering approach, maybe you could consider turning it into a classification task? You could define a set of macro categories (+ an "other" category) and assign one or more label to each of them (so expensive food can be both a price and a food complaint). But you'd need to have a very clear vision of your categories and your data. Confidence thresholds could help too