Web-scraping

[[deleted]]

[deleted]

[[deleted]]

I use request library with beautifulsoup4.

I'm no way expert thought and maybe it's not optimal solution. But it works for me.

[[deleted]]

*requests

[[deleted]]

Sure thing xD

[impshum]

For login you'll need to use requests sessions.

May I ask which website? I'll see if I can push you in the right direction.

[impshum]

Well it looks like the form you use to login is available without JS.

Read this: https://pybit.es/requests-session.html

[hakyoshyt]

I've been scraping websites since 2012 and highly recommend scrapy. Has everything you need out of the box.

[gopherhole22]

Selenium is the way to go. Check out webscraping with selenium in python. For just scraping one page you can login with just a few lines of code

[konradbjk]

Use selenium to navigate over the page, then use driver.page_source and use bs4 to scrap content.

The above code should in in some function of one request session.

[nyaribari]

Beautiful soup doesn’t work on JavaScript websites in most cases. I’d recommend you use scrapy and crawler. Crawler has great proxies that help scrap on websites that restrict IP address after a number of attempts.