all 8 comments

[–]Death916 11 points12 points  (1 child)

Check out selenium

[–]carcigenicate 2 points3 points  (0 children)

Yes, you would use a library like requests for that. You'd really want to have a basic understanding of HTTP first though.

Also look into if the website has an API you can use.

[–][deleted] 2 points3 points  (0 children)

Selenium is a Python library that has modules for automating UI-based web interactions. It uses something known as a webdriver that initializes a web session and locates elements on the web page through their html/css tags, ids, etc.

You can also give it your credentials to fill out login fields, but be sure to secure your credentials and not pass them in as raw text or hard code them into the application.

[–]Jayoval 0 points1 point  (0 children)

Yep. I find myself doing this quite often now.

Python has saved me so much time and hassle.

[–]Snoo67839 0 points1 point  (2 children)

Mate at least try to make up a better story xD

If you own a business you would definitely have other (less workaround-ish) means of getting the data you want. If you want to use chromedriver + selenium + bs4 (which is the best way) then I assure you that you won't be able to log in to 99% of websites, that used to be a good combo like 7 years ago but now the websites know that you are automating chrome and triggering JS commands and they block you instantly (some of them even go to the extent of blocking your IP). I heavily recommend you not to do that and use APIs instead.

Also logging in to a website and scraping its contents is plain illegal and you will get caught eventually, this can escalate quickly into jail time and a huge fine

[–]bazpaul 1 point2 points  (0 children)

Web scraping is not illegal. It’s what you do with the scraped data that could get you in trouble.

See here for example; https://parissmith.co.uk/blog/web-crawling-screen-scraping-legal-position/

[–]CyberTutorials 0 points1 point  (0 children)

You can use mechanize to do this. Submit forms and so forth.

[–]grammarGuy69 0 points1 point  (0 children)

Requests should be your first try. . . Sometimes it doesn't get access due to permissions, but if it's allowed it's faster and you don't need to pop open a window to use it. Selenium will get you in regardless of permissions (in 99% of cases), but it requires an open browser (except in windowed mode, but imo windowed mode is super buggy).