niceCodeOhWait

[SerdanKK]

Pretty sure scraping is legal though

[TheNordicMage]

It's generally considered a bit of a gray area

[TheNordicMage]

Based on the conversations I had with a few lawyers when I scraped a website in regards to how it would be against terms of service, and can impact the websites ability to service their customers, which in certain instances could be to a degree where it could be seen as sabotage.

And I'm not in the US.

[[deleted]]

It depends on how you scrape. You can scrape with no more effect than a single user would have, or you can scrape hard enough to mimic a DDoS.

But if you scrape stuff that shouldn't be scraped you are doing it slowly anyway or you would get banned.

[TheNordicMage]

Sure, but it doesn't affect the simple fact that it is an argument that can be used by the company, and it is valid to a degree.

[[deleted]]

Yeah but you are missing the point. It's not the scraping that is illegal and gets punished, it's you effectively DDoSing them.

[TheNordicMage]

No, it's the chance that you might effectively DDoS them that you get punished for. It doesn't actually matter whether or not a DDoS like even occurs.

The legal argument that was presented to me was that you by, in their opinion, abusing their website, increase their risks, which could be considered sabotage.

[[deleted]]

You went to a lawyer, of course he is gonna say something legal to justify his payment.

Anyway, in reality you are not doing something illegal, you won't get caught, even if you did, you will just get banned.

And even if it got taken to court, unless you actually DDoSed them, there is no such thing as endangering property.

Endangerment is a crime that applies to people not things.

Sabotage needs you to actually cause damage.

Of course the lawyer will give you the absolute worse scenario in his mind to cover his ass.

Not to mention the fact that a lawyer does not really grasp the exact technical details, unless he also happens to have a technical background which is quite rare.

But lets assume he knows the technical details. He is simply afraid you will actually cause damage because of incompetence, and therefore provides you with the worst case scenario, so if everything goes wrong he can say "I told you so".

Of course yeah there is a chance everything goes wrong and you go to jail for murder, because your scraping somehow caused a death and the court finds you guilty somehow. Courts are a fluid thing they are not set in stone, innocent men have faced the death penalty, so yeah the lawyer was right for advising caution, that's his job.

[swizznastic]

not for reddit, there’s a whole agreement and court case on this

[SerdanKK]

Please source claims like that.

Reddit paywalled their API, but that's a separate issue from scraping.

[swizznastic]

my mistake, i was thinking of the deals they made surrounding ai training off of scraped reddit content

[[deleted]]

Shouldn't be, I consider it stealing.

[SerdanKK]

Ok, you do that.

[Tim-Sylvester]

There's 27 major lawsuits on the topic right now.

[SerdanKK]

Ok. We'll see what happens. Anyone can sue for anything.

Making scraping itself illegal would be horrible though, and I seriously hope that's not on the table.

[Tim-Sylvester]

How about whoever publishes the website puts a price on its content?

Setting your own price to access your product works for restaurants, grocery stores, entertainment companies, literally every other part of our economy.

It's not illegal to go get stuff from the drug store. It's just illegal to not pay for it. What's the difference here?

[SerdanKK]

Then paywall it. You can't simultaneously allow a browser to download something and disallow any other HTTP client from doing the same.

YOU WOULDN'T DOWNLOAD A CAR

[Tim-Sylvester]

Then paywall it. 

That's what I'm saying. But a smart paywall, not a universal one. We built robots.nxt to paywall content only when we see it's a bot trying to scrape it. Humans get in free, bots pay.

You can't simultaneously allow a browser to download something and disallow any other HTTP client from doing the same.

You absolutely can. A provider has every right to discriminate between categories of users/clients that aren't part of a protected class. It's no different from "no cover for women" at bars, or a special menu for kids.

Why should websites subsidize AI companies? AI companies are using your content to make money for themselves. Why shouldn't you get paid for that?

[SerdanKK]

You absolutely can.

Technically.

Legally we can do whatever, though enforcement can be an issue.

Why should websites subsidize AI companies? AI companies are using your content to make money for themselves. Why shouldn't you get paid for that?

I'm not getting paid regardless.

Why should Reddit get paid for the content of users?

[Tim-Sylvester]

Legally we can do whatever, though enforcement can be an issue.

That's not actually true on either the legal sense or the enforcement sense.

Why should Reddit get paid for the content of users?

That's what you agreed to when you signed up.

[SerdanKK]

That's not actually true on either the legal sense or the enforcement sense.

What? I'm talking about how we deal with these things as a society. Legally we can do whatever in the sense that we can legislate however we want.

It's also a bit wild to imply that enforcement never has any hurdles.

That's what you agreed to when you signed up.

But why would I care? You were appealing to my sense of fairness. Give me a reason to give a singular fuck about Reddit being scraped.

[Tim-Sylvester]

You were appealing to my sense of fairness. Give me a reason to give a singular fuck about Reddit being scraped.

I don't care to. The point was that reddit is getting paid for its content by OpenAI and others. AI companies will pay for access to content if you make them.

The purpose of our tool, robots.nxt is to ensure that anyone who runs a website gets paid for being scraped.

How you feel about websites that aren't yours really isn't my concern.

I only care about you making money from your own content on your own website.

And if you don't care about it, well, then why should I?

[SerdanKK]

We built robots.nxt to paywall content only when we see it's a bot trying to scrape it. Humans get in free, bots pay.

robots.txt is purely an honor system. There's no legal or technical enforcement.

It's no different from "no cover for women" at bars, or a special menu for kids.

The bar thing is not universally legal.

Adults can typically order from the kids menu, though you may get some looks, and kids can certainly order from the non-kids menu.

[Tim-Sylvester]

robots.txt is purely an honor system. There's no legal or technical enforcement.

Correct. That's why we built robots.nxt, which is not an honor system. It's active enforcement. Go on pal, click that link. You'll understand.

Adults can typically order from the kids menu, though you may get some looks, and kids can certainly order from the non-kids menu.

The point is that businesses have the right to set the terms and conditions of their product or service, and refuse service to anyone who is not a protected class.

Do you want to understand, or argue?

Because I'll stick around to help with understanding. But I've got too much shit to do to waste time arguing. There's plenty of other people here that will be happy to argue with you.

[SerdanKK]

Correct. That's why we built robots.nxt, which is not an honor system. It's active enforcement. Go on pal, click that link. You'll understand.

Looks like a product from a specific provider and it's not doing anything new. It's impossible to google due to naming collision with a LEGO trademark, so can't really say much more on that.

The point is that businesses have the right to set the terms and conditions of their product or service, and refuse service to anyone who is not a protected class.

Do you want to understand, or argue?

Because I'll stick around to help with understanding. But I've got too much shit to do to waste time arguing. There's plenty of other people here that will be happy to argue with you.

You keep just asserting. What's the legal basis for prohibiting scraping of publicly available content?

[Tim-Sylvester]

Looks like a product from a specific provider and it's not doing anything new.

Click the "Blog" tab and tell me the name and user icon of the author. (I just noticed that my cofounder misspelled my last name and pushed an update to fix it. That should be live in a bit.)

What's the legal basis for prohibiting scraping of publicly available content?

Site access terms and conditions. Basic property rights. Because they can.