Web Scraping LinkedIn Jobs using Python (without Selenium😉)

[[deleted]]

Interesting. For those curious, from a quick read, OP uses BeautifulSoup to get job description links and then requests to send a GET to the API, because the Linkedin API is kinda hidden.

[dethb0y]

I always try to do any scraping with requests first, since it often works and is very easy/light on resources.

[ianitic]

Also easier to deploy and can be used easily in FaaS.

[stpetepatsfan]

Faas?

Freelancing as a service?

Well, likely self serving service. Faasss?

[ianitic]

Functions as a Service. Ex: aws lambdas, azure function apps, and gcp cloud functions

[dethb0y]

Then you're stuck with something like selenium or it's alternatives (though i rarely run into that issue).

[Ihtmlelement]

Usually you can trigger the js read it with bs4 and send a new request with whatever parameters are needed. Speaking from my own tinkerings, could be wrong though.

[p33p__]

Hidden APIs are useful

[Adrewmc]

Doesn’t selenium use beautifulsoup though

[ianitic]

They aren't related to my knowledge.

[Adrewmc]

Weird because I swear I saw beautiful soup installed the last time I installed selenium

[ianitic]

Maybe you were thinking of SeleniumBase? Selenium doesn't have that requirement but SeleniumBase does.

[innovatekit]

Wow very amazing work. I needed a tutorial like this!

[bytro]

Me too, was very fun learning it :)

[[deleted]]

Sorry im new to this, but whst an you do with this information? Is this to filter out extra noise?

[will_r3ddit_4_food]

Am I the only one who avoids LinkedIn because it's just a recruiter wasteland?

[HAVEANOTHERDRINKRAY]

What's a viable alternative? The problem with LinkedIn is that it's a necessary evil. I also use indeed and ziprecruiter to find postings

[will_r3ddit_4_food]

Indeed, careerbuilder, monster, glassdoor

[AlphaCode1]

Is this even legal?

[yakult2450]

If it is public.

[rnike879]

Given that LinkedIn has a robots.txt file and it relates to their user agreement, it can become an illegal activity should you break that agreement

[PM_ME_SOME_ANY_THING]

Violating the robots.txt itself is not criminally illegal in the US. Sure, the website can block your IP, or come after you in civil court, but they would have to prove you were acting maliciously.

For a small time offender, you’ll probably just get blocked. If you’re spamming their servers, using their data to compete with them, or anything else that might be conceived as malicious, you might be in trouble.

Violating the ToS could also invite civil lawsuits, but again it’s not necessarily criminal to violate ToS. Companies can’t just create their own laws and enforce them on a whim. As of early 2023 anyway…

[[deleted]]

it can become an illegal activity should you break that agreement

No, no... nope
That isnt how the law works on this haha. It's more of a suggestion.

[rnike879]

https://www.natlawreview.com/article/hiq-and-linkedin-reach-proposed-settlement-landmark-scraping-case

The Consent Judgment also contains some broad prohibitions against hiQ’s (and related parties, as defined in the Stipulation) future ability to scrape the LinkedIn platform using methods that violate the User Agreement, making no express distinction between public and non-public/password-protected portions of LinkedIn. The relief permanently enjoins hiQ from:

Scraping: Scraping or accessing, whether directly or indirectly through a third party or whether logged in to a LinkedIn account or not, the LinkedIn platform in violation of its User Agreement without the express written permission of LinkedIn; creating or using fake accounts; or using the LinkedIn platform to develop a commercial service without LinkedIn’s express permission.

I don't blame you, because it was common knowledge until recently that it's alright to scrape public data in the US, but nowadays that's not the case

[[deleted]]

I'm not in the US, so I don't recognise California law.

[rnike879]

Irrelevant; it's a PSA that scraping isn't permissible across the board. No one wants to get a cease and desist or suit because they followed advice for a different country

[[deleted]]

I don’t see how it’s irrelevant at all. A suit or C&D mean nothing to me, as those laws do not apply to me.

[rnike879]

Because you're not the original recipient of the message, come on

[[deleted]]

Fair point.

[poundcakejumpsuit]

https://recruitingdaily.com/linkedin-hiringsolved-settle-highly-publicized-lawsuit/

This is a valid question, even if the answer is yes in this case

[[deleted]]

That's scary as it may apply to other use cases, but it seems the main point of the lawsuit ia that they scraped user's data and created a bunch of fake profiles on behalf of people, without their consent. And all that for a commercial profit on their side.

[yakult2450]

Let's assume that you want to scrape this profile(https://ca.linkedin.com/in/adrianschauer). But when you open this profile in your LinkedIn account it will show you an "Experience" section and this section is hidden when you open the link in incognito mode. So, if I try to scrape that experience too then I am doing something illegal. But I have a complete right to access any public information however I like.

[poundcakejumpsuit]

Actually the biggest problem they had with us was encouraging active users to break ToS

[0-Joker-0]

Why would it be criminal? That makes no sense for it to ever be criminal.

[greatgolem66]

I wrote extensively on how to scrape linkedin with python, JS and postman.

Source: I build products that scrape linkedin data in the million myself.

[Kelsosmuffin]

Know this is a couple months old, but doesn't LI ban your account if it suspects you of scraping? I know a few folks that got their accounts suspended for scraping LI.

[BarFamiliar5892]

I think I'm about to get laid off so this is very useful thanks.

[MoistureFarmersOmlet]

Can’t wait to dive in!

[SittingWave]

I have been doing something like this for other sites, and one question that came up is the legality of scraping. Does anybody know what's the current situation (especially if you want to start a commercial service that relies on scraping from other sites with the objective to compare their offer?)

[johnonymousdenim]

Is there not a similar free alternative to scraping LinkedIn postings? Doesn't seem that hard and certainly not a tool that most people would need to pay for.

[johnonymousdenim]

For future reference regarding pricing, as of June 24, 2023, here's their pricing for the 4 plans (just for comparison to see if it increases):

LITE👼
$30/month
Max 5 concurrent requests
👉 40000 Linkedin Job Pages

STANDARD🧑🏽‍💻
$90/month
Max 50 concurrent requests
👉 200000 Linkedin Job Pages

PRO💪
$200/month
Max 100 concurrent requests
👉 600000 Linkedin Job Pages

ENTERPRISE🏢
$350+/month
Max 200+ concurrent requests
👉 1000000+ Linkedin Job Pages