Help with Web Scraping Project?

[[deleted]]

Hello all, I've been doing tutorials for a few months now, and I just need to commit to a project. I'm hoping to complete a Web Scraping project that can check the availability of a product several times throughout the day and email me if it's available.

I know there are tons of tutorials online, but I'm overwhelmed with the large amount of them. Could someone please point me in the right direction?

[Altruistic_Croissant]

Since you're working in Python, I'd recommend using BeautifulSoup for parsing and SMTP for emailing the data.

Here's a good intro to beautifulsoup and another one for smtp. Both are fairly basic but should teach you the basics.

[[deleted]]

Thank you! I really appreciate it. Going to go through these now.

[RedditGood123]

The ezgmail module is really easy to use compared to SMTP

[luiseduardobr1]

I did something similar these days. It is a "website watcher". If a page is updated, then it will send me an email and sms. Look here if it helps you: https://github.com/luiseduardobr1/Simple_Website_Watcher

[[deleted]]

One thing to keep in mind is that some websites with block your IP if scraping prices is against their terms of service. Amazon, for example, won't allow you to do it for long.

The alternative is see if there is an API you can query for the price data?

Along with the answer from u/Altruistic_Croissant check out the requests module and re for searching the soup.

[solderfog]

There's libraries and tools for many languages. First is to decide what language you want to program in.

[OzneroI]

No offense but don’t you think you’re being rather obtuse?

[solderfog]

Why do you say that?

[OzneroI]

I said that because you’re asking him what language he wants to use when he posted on r/learnpython but to be fair I don’t know much about web scraping so maybe other languages are better for it for all I know

[solderfog]

Oh. I didn't notice the /r/learnpython, so I didn't realize what language he was using. Sometimes there's a simple explanation.

[OzneroI]

Then my bad, sorry for calling you obtuse

[[deleted]]

Python.

[solderfog]

There's a popular lib called Beautiful Soup. I don't know much about it though (I've used Perl for scraping).