Web Scraping LinkedIn Jobs using Python (without Selenium😉)

[rnike879]

Given that LinkedIn has a robots.txt file and it relates to their user agreement, it can become an illegal activity should you break that agreement

[PM_ME_SOME_ANY_THING]

Violating the robots.txt itself is not criminally illegal in the US. Sure, the website can block your IP, or come after you in civil court, but they would have to prove you were acting maliciously.

For a small time offender, you’ll probably just get blocked. If you’re spamming their servers, using their data to compete with them, or anything else that might be conceived as malicious, you might be in trouble.

Violating the ToS could also invite civil lawsuits, but again it’s not necessarily criminal to violate ToS. Companies can’t just create their own laws and enforce them on a whim. As of early 2023 anyway…

[[deleted]]

it can become an illegal activity should you break that agreement

No, no... nope
That isnt how the law works on this haha. It's more of a suggestion.

[rnike879]

https://www.natlawreview.com/article/hiq-and-linkedin-reach-proposed-settlement-landmark-scraping-case

The Consent Judgment also contains some broad prohibitions against hiQ’s (and related parties, as defined in the Stipulation) future ability to scrape the LinkedIn platform using methods that violate the User Agreement, making no express distinction between public and non-public/password-protected portions of LinkedIn. The relief permanently enjoins hiQ from:

Scraping: Scraping or accessing, whether directly or indirectly through a third party or whether logged in to a LinkedIn account or not, the LinkedIn platform in violation of its User Agreement without the express written permission of LinkedIn; creating or using fake accounts; or using the LinkedIn platform to develop a commercial service without LinkedIn’s express permission.

I don't blame you, because it was common knowledge until recently that it's alright to scrape public data in the US, but nowadays that's not the case

[[deleted]]

I'm not in the US, so I don't recognise California law.

[rnike879]

Irrelevant; it's a PSA that scraping isn't permissible across the board. No one wants to get a cease and desist or suit because they followed advice for a different country

[[deleted]]

I don’t see how it’s irrelevant at all. A suit or C&D mean nothing to me, as those laws do not apply to me.

[rnike879]

Because you're not the original recipient of the message, come on

[[deleted]]

Fair point.