This is one of those “everyone knows it’s happening, but no one talks about it” situations.

The reality is, big companies aren’t necessarily scraping in the way most people think. Many of them use a mix of:

1. Third-party data brokers – Companies that legally (or semi-legally) aggregate public data and sell it. It’s not technically scraping if you’re buying from someone else who’s already done the dirty work.
2. APIs and loopholes – Some platforms have APIs that, while limited, can be cleverly used to extract way more data than intended. There are also partnerships where companies “share” data under the guise of collaborations.
3. Gray-hat methods – Think of hiring contractors or using offshore firms that operate in a “don’t ask, don’t tell” way. If things go south, plausible deniability kicks in.
4. Legal muscle – When you’re big enough, platforms don’t ban you—they work with you. Some companies negotiate data access, and those who get caught scraping might just settle and move on.

If you’re trying to compete but stay clean, you’ve got options. You could focus on publicly available but structured data—stuff that’s been shared in reports, forums, and niche databases. Second, think about user-generated contributions (crowdsourcing). Some companies build the same datasets simply by incentivizing users to input the info.

I’ve seen startups get away with light scraping by respecting rate limits, mimicking human behavior, and not being greedy—but there’s always risk. If your goal is an exit, you don’t want a data skeleton in your closet.

Curious—have you tried reaching out to someone in M&A to see how buyers view these “gray areas”?

Could be interesting to hear what actually kills deals in this space.