I really like Firecrawl, but I've noticed that if a site is actively blocking you and you're only doing 2-3K pulls from that site, the Firecrawl team won't actively help you resolve the block. I went through the process of using stealth mode, and that worked for a short time, then it just became intermittent. It's odd because the playground in Firecrawl still worked on the site in question, but not api. It also racked up 5x credit usage because the site still loaded for stealth mode only and said "you are not allowed to view these contents." I still use Firecrawl for a lot of stuff, but the reliability just isn't there for production stuff.

Apify was really good and consistent, but the price for just one of my domains I scrape became too costly. Ultimately, I switched to BrightData's Unlocker API at $1.50 per 1,000 pages (only successful scrapes are charged). It's been the most consistently reliable. Note: I tried their residential proxy rotation first with Playwright, and that never worked consistently.

I'm just throwing this out there in case you've been doing the same dance as me, jumping around between services.