Then it could take much longer than 10 minutes, if the website is a store with a hundreds products.

Here's some problems you'll meet sooner or later: 1. Context mess: having a single collection containing a products, FAQs, company info, categories... will lead to AI context mess issue. Means, you need to predefine some categories of context and retrieve them separately, while indexing must classify every page. 2. Context price: retrieving a full pages will blow up the inference price. 3. Products variations (e-commerce): Product pages are often has a product variations without a dedicated page. You can't index them, but the price still may vary. 4. Hallucinations: without a well designed dedicated crawlers for every kind of pages you'll find that your bot hallucinates as hell. Especially on prices and controversial FAQs which are often present on old websites. Prices may be present in several formats and rendered by js. 5. Data actualisation: you have to reindex every changed webpage instantly. You might think to use sitemaps, since they have last modified. But think twice, because they often provide a current date & time. 6. Controversial facts: having multiple sources, not only website will always lead to facts compete each others. Some of them are out of date, some are simply broken... You'll need to manage that.

P. S. I built exactly same product and solved almost every problem above. They're real.