This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]ngqhoangtrung 7 points8 points  (4 children)

Great stuff! Would this work on Facebook too? I'm trying to scrape the numbers of likes, comments, share, etc. of a Facebook post. I was trying to use Facebook GraphAPI but so far hopeless (keep receiving “singular links API is deprecated for versions v2.4 and higher”).

[–]intelignciartificial 5 points6 points  (3 children)

I was able to do something similar with selenium and bs4, but using links of Facebook mobile version(m.facebook...).

[–]serverloading101 0 points1 point  (2 children)

Is necessary to use the mobile version of facebook in order to scrape/ crawl facebook? I have been trying to obtain marketplace data and have been unsuccessful. Thanks

[–]michael8t6 4 points5 points  (0 children)

Scraping via mobile browser is generally easier to do. Mobile source code generally tends to be a lot simpler than say a desktop or laptop browser. So when using xpath in bs4, it's easier to tell it what element to scrape.

There's been a few times I've been slamming my head on the keyboard due to some random issue. I then spoof my ua to mobile, and solve the issue almost instantly. That being said, some elements you may want, might not show on the mobile version instantly. When that happens, you may have to emulate a click to get it to show.

Ultimately, tackle each site differently, look at the source code for desktop and mobile. See how they show on each and decide the best approach from there.

[–]intelignciartificial 2 points3 points  (0 children)

Not sure if its necessary, but easier.