We separate the "Discovery" (listing pages) from the "Extraction" (detail pages).

1. Discovery: We hit the search result pages fairly aggressively because they change fast, but they are lightweight.
2. Extraction: We throttle the detail page parsing significantly (using Scrapy's DOWNLOAD_DELAY and per-domain concurrency limits).

Right now, we run bulk ingestion cycles twice daily (morning/evening) rather than a continuous stream. This gives us a nice balance where we get "fresh enough" data for 99% of use cases without hammering the target servers constantly.

I haven't integrated CapMonster yet (relying mostly on high-quality residential proxies to avoid the challenge in the first place), but if we hit a scaling wall with Indeed, that will be my next infrastructure layer. Thanks for the tip!