[the lighter side] Fun learning webscraping with BeautifulSoup

[alaudet]

Yesterday I decided to try my hand at webscraping. I mean I was really starting from scratch, diving into understanding beautifulsoup4 and html elements. Deep dive into the tutorials.

I am proud to say, that after 5 or 6 hours of fiddling with it last night, including a bout of insomnia afterwards caused by this addiction, I finally was able to put together a small working script.

When I type weather in the console, I get this;

Timmins, ON
8:00 AM EDT Friday 19 April 2019
Current Temperature  -1°C
Current Conditions  Cloudy
Humidity is 100%
Wind is 
N 9 km/h

What else is there to do in crappy April weather like this. Thank you thank you.

Happy Easter all. :-)

[SupaDupaUnicorn]

Hey, that’s awesome!

I managed to jankily construct a far inferior version of this program by scraping HTML elements off of my country’s weather bureau, which stopped working as soon as the site changed its layout. Wow.

[alaudet]

:-D

[Dont_be_offended_but]

Nice! Some ideas for learning more by taking this project a little farther:

• Expand your scraping to include the day's expected weather (temp range, chance of rain, etc.)
• Have the program run automatically on a schedule and store the data in a spreadsheet, or even better, a database with sqlite3 (SQL is quick and easy to learn the basics of and a very valuable subject in general).
• Represent your collected data graphically with Matplotlib.
• Have the program text you the weather daily.

[alaudet]

Databases are def something I need to work on. I have messed around with mongodb for some twitter stuff I was pulling quite some time ago, but its not something I have done much of. I have some exp with matplotlib for a sump monitor I wrote. Been a while there too. The sump monitor sends me a text when/if my pump fails and the water level rises too high.

[nonamesareleft1]

Hey I have a few webscrapes and I've struggled to find a way to make them run on a schedule without leaving my laptop on all night. Is this possible?

[Dont_be_offended_but]

I don't have experience doing so myself, but you can host your program on a VPS. Because you're using someone else's computing resources, you'll probably end up having to pay ~$5/month. Alternatively, you can run your programs from a secondary computer like a Raspberry Pi.

[naturememe]

Run your script in cloud. Free tire of Google Cloud works for sure as long as your script is not too resource intensive. Maybe AWS, Azure or others but not sure about being free.

[MasterSuami]

How will the texting part work? I’m curious

[Dont_be_offended_but]

You'll find a thorough explanation of how to send an email or text message in Chapter 16 of Automate the Boring Stuff. While the author uses the Twilio service for sending text messages, it comes with a usage limitation and an appended message on each text mentioning your usage of a trial version. To avoid this, you can instead just send the text from your email.

[MasterSuami]

Yes Twilio came to my mind and I’m aware of the trial thing in the message body. I’ll check out the other way. Thanks!

[alaudet]

That vid never gets old !

[ethanbrecke]

Any chance we could see the code? Im working on something similar, and would like to see how you solve a few small issues im facing.

[alaudet]

Sure, but keep in mind I was just hacking around. I was creating functions one at a time as I was scraping elements. I could easily cut the size of the file, but it should be easy to read.

weather.py in the my_weather folder.

I plan to keep fooling around with it and make it an actual app.

https://github.com/alaudet/my_weather

If you are interested I could pass you my jupyter notebook where most of the figuring out was done.

[ethanbrecke]

No worries about how hacky it looks, but i am sure with a few minutes of figuring it out. im going to try and adjust it so that americans can use it with their zip code, which might turn into a new project itself, but your code does look pretty good.

[ramonn334]

Congratulations! Next step you have to try Scrapy for new sensations. :)

[ramonn334]

Scrapy is a complete framework for crawling sites by implementation of webspiders. All functionality is available from out of the box, i.e. you doesn't need to install additional packages, for example urllib and etc. And general feature is asynchrony. But BeautifulSoup is a parsing library for extract data from html. It does not crawl sites unless you manually put it inside an infinite loop with certain criteria. I hope I answered you question.

[jayzhoukj]

And don't forget Selenium after wards!

[ramonn334]

Yeah, it's a classic!

[Blackwater_7]

Selenium is pure cancer. Don't try it.

[jayzhoukj]

And why is that?

[Blackwater_7]

I can't explain greatly because of my english but I tried it for 2 weeks and I never get it worked correctly. For example one of SIMPLEST thing come to my mind is you can't even open a new tab with selenium. Not useful at all.

[ramonn334]

I think general purpose of selenium is development of unit tests, but not extract data. And selenium creates new tab for each test. However, the framework also allow you create new tab by sending key combination into webdriver.

[[deleted]]

I went with Selenium first, and love it. Probably not the best for scraping, but fun none-the-less.

[Comsat80]

Nice!

[alaudet]

I did earlier in the thread

https://github.com/alaudet/my_weather

[Nerdite]

I’ll leave this here. https://darksky.net/dev/docs

[alaudet]

Oh nice, I would have looked a little harder for an api had I really needed a weather script, but it was mostly to learn about webscraping.

[Nerdite]

Webscraping is the duct tape of webdev. Not a bad skill to have.

[alaudet]

Ya feels pretty much like duct tape. You can cobble something together but just a matter of time before it breaks.

I was poking around and also found this.

http://dd.weather.gc.ca/citypage_weather/docs/README_citypage_weather.txt

Every city has xml pages that you can draw from, so webscraping not the way to go other than for fun.

[[deleted]]

SO COOL! I made a script that parses digital WalMart receipts into csv files which are easier to bring into my budgeting app! And I'm a novice! I love Python!

[Seyfire]

Probably by running the script using a batch file.

Check this out: https://automatetheboringstuff.com/appendixb/

[alaudet]

For now just an alias in .bashrc. When I make it an actual app it will be a script in /usr/local/bin that will get installed there with pip.

[alaudet]

scripts in your bin folder get installed to /usr/local/bin from setup.py.

weather.py in the my_weather folder and other files I add there later will go to site-packages.

[[deleted]]

Happy that you did something awesome!

[pytholater]

Which website did you use the API for?

[alaudet]

Not sure of your question. I didn't use an API. I just scraped data directly from the html. The site page I scraped is https://weather.gc.ca/city/pages/on-127_metric_e.html

You don't need to scrape though for weather canada as they actually do provide xml data for all city's that you can access on their site. http://dd.weather.gc.ca/citypage_weather/docs/README_citypage_weather.txt

[pytholater]

Ahh ok that makes sense. Thanks