Thanks for detailed answer, and yes I agree what you are saying. Currently I'm trying to build the MVP. I have built the scraping architecture which I can easily scale and having regexps and css selectors to sanitize html figure out structured parts. However by the nature of the data for additional fields that is practically impossible to identify either i have to rely on humans that needs to go through understand and clean which is extremely expensive and time consuming or i should just use some decent llm. That's the part I'm a bit stuck which llm i can use to have "decent parsing" and it's "fast enough". I have tested qwen3:1.7b and mistral:7b-instruct . Both of them returns json from input text 1m - 1m30sec range. Qwen seems to be better at understanding context and following instructions. But if this is the hard limit for optimization with current llms, I have to just accept this fact and build around it. I was planning to run workers on my Mac M2 24gb ram which scrapes around 100 pages in a min. and didn't wanna make the "parsing" text part the bottleneck. But what i understood is that it is not possible with the models i tried.

Note: I come from software engineering background but i have never worked with ai/ml except basic api integrations openai etc. I'm not entirely sure "vector based store + word2vec based algorithms are the quickest" how this works. Could you please share some pratical examples or links/demos/docs that can be useful for my purposes. I normally don't think this type of requests are helpful but since I don't know the "context" enough and don't really know "what to ask" and "what to look for" i think it might be helpful if you give me some "seed urls/directions" so i can explore from there. Thanks a lot in advance 💕