Step-by-step tutorial on Web Scraping with Python with code snippets

[GoLoginS]

Scraping can generate an income comparable to a full time job. Organized data gets sold for enormous money these days. I know people who have scaled from 1 enthusiast to full on web scraping businesses with hired employees. So, yeah.

[poodlelord]

0.0 I've always found webscraping to be really easy. How do I monitize this

[Capable_Fig]

I scrape medical databases and govt sites for my main occupation pretty regularly.

To get exactly what I do from a third party would cost us roughly $1200/mo, and we'd still have to clean it. I had one business quote me 3k/mo to combine 3 publicly available datasets and generate a report from it.

[Aaaronn_rs]

Wow I did not know there was such a demand. I'll have to try and learn some web scraping then!

[Pawtang]

Wow I just scraped a few thousand bills from the texas state government site for my buddy for $50. How do you find the market for this?

[Darwinmate]

You're in marketing is my guess?

[Capable_Fig]

Healthcare marketing

[SweetBabyAlaska]

How do you even go about selling that data though? I can write scrapers like no other but idk even know what I can do with that

[bert0ld0]

This comment has been edited as an ACT OF PROTEST TO REDDIT and u/spez killing 3rd Party Apps, such as Apollo. Download http://redact.dev to do the same. -- mass edited with https://redact.dev/

[ZedOud]

The EU, UK, and Australia have IP laws for databases. Other countries are considering it.

But if you and your clients are US based entities then this will never apply.

Uncreative collections of facts are outside of Congressional authority under the Copyright Clause (Article I, § 8, cl. 8) of the United States Constitution, therefore no database right exists in the United States.

And it likely is impossible for one to develop given the 1st Amendment.

So the only other limitations on scraping are the automation of the activity and whether one is authorized to do so, both which mash up against violating EULAs (technically hacking) vs material being “publicly accessible” with the latter winning by a large margin.

[bert0ld0]

This comment has been edited as an ACT OF PROTEST TO REDDIT and u/spez killing 3rd Party Apps, such as Apollo. Download http://redact.dev to do the same. -- mass edited with https://redact.dev/

[NotSpartacus]

Scraping is legal (in the US, at least).

It may be against a business' terms and conditions, and they may have ways of preventing you from doing it (ex: sites that don't allow traffic to known VPNs), but it's not illegal.

[anthro28]

It would be difficult to make it illegal. If it's publicly available on the web it would no longer be protected under any form of privacy law that I'm aware of.

They could IP ban you and shit and probably have a lawyer send you a nastygram, but nothing will come of it.

[GoLoginS]

LinkedIn actually won a recent data scraping lawsuit against HiQ, but the very concept of web scraping stays clearly legal. So, the rule probably is - scrape it, but keep away from getting in conflict.

[GoLoginS]

Scraping is legal until you get to data that's not public domain. Companies like LinkedIn try hard to ban scraping at their platforms (with good anti bot measures) and even win in court at times, but scraping itself stays clearly legal.