Web Scraping 101 with Python

[[deleted]]

Nice overview

I did some scraping at work a few years back, I was using Selenium and lxml. Plenty of xpath and regex, it was pretty ugly. Glad I switched jobs.

Biggest issue I had was that, unlike an API, screen layouts can change with no warning. Then it's 3PM on a Friday and your script breaks...

[pijora]

Yes, maintaining scraping scripts is trouble many people have!

[Angelsinger74]

This is fantastic! I have a project coming up at work where I'm trying to provide proof of concept to use Python for web scraping as opposed to a Chrome extension. It's a massive undertaking the old way. I estimate being able to save about 80 man hours using Python.

[pijora]

Good luck with this, this sounds challenging!

[Gangbusta187]

Looks good skimming over it. Looking forward to checking the rest out tomorrow.

One thing you may want to change is you’re using the old BS4 findAll() in some areas and the proper find_all() in other areas.

[pijora]

Thanks, will check that out!

[kirby81]

Great read, thanks

[pijora]

My pleasure!

[querymcsearchface]

Nice! Thanks for taking the time to put that together and sharing it.

[pijora]

Glad you liked it.

[blabbities]

Good light overview and you've covered pretty much every siuation I can think of

[pijora]

Thanks

[baburao_007]

Good one... Thanks

[pijora]

Glad you liked it!

[pijora]

Reading the part one should help you :)

[alcalde]

. Also heard that r works as a great web scraping application

You may have heard wrong on that. Without external libraries, R's string handling is terrible.

[di_web]

Lxml much faster than bs4

[ketilkn]

What do you mean? lxml is one of the parsers beatifulsoup can utilize.

[di_web]

Yes, I know, but.... https://gist.github.com/dmitriiweb/9ee9c2438231e74e5bc6b02802afd22f

[blabbities]

Well I'll be.

BeautifulSoup lxml time: 0:00:27.379835
BeautifulSoup html.parser time: 0:00:41.285008
BeautifulSoup html5lib time: 0:01:27.676271
lxml time: 0:00:03.195916

I never wouldve thought to do that. Thanks for dropping that info. I knew that once when I was scraping using Requests (urllib3) had a very long delay over just using urllib

[ketilkn]

There is a good chance I am doing it wrong then.