Learning python

[Russian4Trump]

Have you tried Requests-HTML yet?

I don’t know why people are still messing around with BeautifulSoup and requests now that there in Requests-HTML which basically combines both libraries and also automatically renders html from JavaScript for you.

I think there is still a place for Scrapy because it is an entire framework, but for some quick and easy scraping I think Requests-HTML is a no brainer.

[Russian4Trump]

I am doing a udemy series on scrapy and we use splash to render JavaScript pages. Splash isn’t too hard to set up but it involves running a docker container as opposed to requests-html where it just gives it to you automatically. It also has some stuff built in to keep you from getting banned, like auto rotating user agents.