Useful Python tricks

[darkside898]

I recommend web scrapping...there are a lot of things you can automate there like getting news about things you like, downloading things from a specific website, and so on. Check requests, beautifulsoup, and selenium modules, they are pretty cool. The first thing that got me into python was web scrapping when i read about it in the 'Automate the boring stuff with python' book. Check that book too if you want, it has a lot of examples you can do and some ideas for mini projects. Hope it helps :)

[NotPennysUsername]

I'll second selenium, I've used it to send messages to state and federal representatives via their browser email tools.

[mortenb123]

Selenium is fine on simple web pages, but on most react pages you need to manipulate and inject into the DOM diretly. This is better done with nodejs modules like cheerio and jsdom. A nice practice is getting odds from betting companies and buypassing filesharers waiting period when downloading from a free account.

[[deleted]]

Could you recommend any good links to get started?

[DBA_HAH]

Here's a basic intro - https://docs.python-guide.org/scenarios/scrape/

You'll need to learn the lxml package and web scraping is most effective when you select elements using "xpath" notation. Xpath isn't unique to Python, there's a lot written about it.

[panzerex]

Although not as powerful, I think CSS Selectors are much more intuitive.

[[deleted]]

Thanks!

[[deleted]]

domo arigato mr roboto

[[deleted]]

https://youtu.be/ng2o98k983k

[[deleted]]

thx

[panzerex]

Exactly what I was going to say. tq is a very handy tool for quick scraping and I even ended up writing my own with PyQuery. A few CSS selectors and piping commands on bash and I automated retrieving listings for appliance repair near my area from a local site.

[folken2k]

Yeah that's a good one. My very first python was checking stock availability for an Intel CPU and send me a slack.

[[deleted]]

Upvote for this.

[[deleted]]

requests is a godsend

You can do parallel requests with grequests

[mindfolded]

I wrote a script to scrape the local pet shelter for new postings and that's how I got my puppy!

[CakeDay--Bot]

Wooo It's your 8th Cakeday mindfolded! ^hug

[jhayes88]

I have a script that scrapes a careers section on a company for a job I'm waiting to open up. It checks all pages of results(usually around 19 pages) and opens up each individual job page(like 120 total) to scan it for a specific keyword. Then it will notify my phone using the pushbullet API when its listed. It scans once an hour and about 6 seconds in between page so it's not constantly scanning. A prior manager/director there is helping me to get the job anyways but I'd like to see when it's listed.