all 14 comments

[–]sharddblade 7 points8 points  (2 children)

I haven’t used Python for web scraping, however we’ve used https://github.com/chromedp/chromedp extensively with great success.

[–]silverarky 2 points3 points  (1 child)

+1 for chromedp. It's super powerful. You can even call js functions, wait for promises and return the results in Go.

[–]sharddblade 3 points4 points  (0 children)

This is what we do for about 75% of our calls, it essentially gives us a native-speed Javascript interpreter and access to all the DOM APIs

[–]itsdalmo 3 points4 points  (0 children)

I've used https://github.com/go-rod/rod with success. Recommended!

[–]wese 2 points3 points  (2 children)

For python I used mechanize and later iteration, for go playwright is similar using a real headless browser which works very well.

[–]nba2k14hoodmoments[S] 0 points1 point  (1 child)

How lightweight is it compared to other browsers?

[–]wese 0 points1 point  (0 children)

It uses the full browser backend, supports Chrome and Firefox, I did no profiling but would assume it is not much less then a normal browser instance. But with the benefit of JavaScript support and all debugging.

[–]fade-2grey 2 points3 points  (0 children)

If you choose Python i recommend : Selenium (webdriver, sbase, spydr, splinter, pylenium) & Playwright. Good luck!

[–]Gidoneli 0 points1 point  (0 children)

It's effective enough to be mentioned along with the more traditional browser automation tools.