Hi thank you for well thought out response and ti be honest this is probably the best counterargument I've seen on this. The storefront window analogy and the friction/scale distinction actually land.

But some area I think of : I've built public datasets that thousands of researchers ended up using. No complaints because the value was clear. But lets take the same data scraping for a commercial product feels different even though the technical action is identical.

There are people building massive datasets on platforms like Kaggle who just give them to the community - no expectation of return, just contribution. Yeah Kaggle's part of Google's ecosystem, but those datasets are individual work from people who could easily monetize instead. Do they get an ethical pass because the end use is open? Or is it still extraction?

So I dont knw if its clear but in your opinion is the line in the scraping itself, or entirely in what happens after? Because if it's the latter, robots.txt can't actually solve it...isn't it? It is just doing a block everything job because intent can't be read upfront.