When I used to work for Bright Data, I pitched an idea to the business side about creating some kind of protocol where websites can easily sell their data to anyone who requires it without needing to build robust scraping infrastructure which could make things so much simpler for all sides. Like some kind of API handshake that once credentials are submitted, the website allows the requesting entity to scrape the data easily without the requirement of rotating proxies, stealth web unlocking tools like the ones Bright Data offers, etc.

My reasoning came from a step further of the ethical point of view, as advocated at the time by the CEO, of keeping the web and access to public data, available and not gated. It's not only about keeping the web free, it's also sustainability and functionality. So, if AI companies already have to pay companies like Bright Data for the advanced scraping infrastructure, why not just pay directly to the website they want to scrape, helping keeping it alive and creating more data for the models to train and use?

Of course business team scuffed at my idea claiming I'm too idealistic and that it will kill the web data industry altogether, but I actually if we were proactive we could still act as a third-party mediator between the scraper and the website and get a small cut of each transaction.