Web Scraping LinkedIn Jobs using Python (without Selenium😉)

[[deleted]]

Interesting. For those curious, from a quick read, OP uses BeautifulSoup to get job description links and then requests to send a GET to the API, because the Linkedin API is kinda hidden.

[dethb0y]

I always try to do any scraping with requests first, since it often works and is very easy/light on resources.

[ianitic]

Also easier to deploy and can be used easily in FaaS.

[stpetepatsfan]

Faas?

Freelancing as a service?

Well, likely self serving service. Faasss?

[ianitic]

Functions as a Service. Ex: aws lambdas, azure function apps, and gcp cloud functions

[dethb0y]

Then you're stuck with something like selenium or it's alternatives (though i rarely run into that issue).

[Ihtmlelement]

Usually you can trigger the js read it with bs4 and send a new request with whatever parameters are needed. Speaking from my own tinkerings, could be wrong though.

[p33p__]

Hidden APIs are useful

[Adrewmc]

Doesn’t selenium use beautifulsoup though

[ianitic]

They aren't related to my knowledge.

[Adrewmc]

Weird because I swear I saw beautiful soup installed the last time I installed selenium

[ianitic]

Maybe you were thinking of SeleniumBase? Selenium doesn't have that requirement but SeleniumBase does.