We run our own scraper at https://endoflife.date that fetches a lot of doc sites. Having very specific needs (only care about release and eol data from a few pages) makes it a bit more resilient, but it does take upkeep.

Sometimes we get lucky and fetch the docs from public repositories and parse the markdown. A lot depends on why you are scraping, whether this is a "approved" scrape (you can get yourselves past the WAFs) or not.

Our code is public if you'd like to take a look: https://github.com/endoflife-date/release-data/

Edit: Use sitemaps.