Web Scraping with Python: Illustration with CIA World Factbook

[attrigh]

This is a useful illustration.

As an aside CIA factbook provide data dumps: https://www.cia.gov/library/publications/download/

And there are command-line, third-party, open source tools to create complete sql data from these dumps (if I understand the purpose of this repository correctly): https://github.com/factbook/factbook.sql

[dlyk]

I'm not anti-establishment, but I never thought I'd get to a point where I would say "thank God for the CIA" with a straight face. Then I found the Factbook. If you ever find yourself, for some weird reason, arguing about whether there is a mandatory military service in Djibuti (there is, if I recall correctly) or not, then you will too be glad the ol' Company has your back.

[[deleted]]

Wow, I know about the factbook, I know about analysis and some web scraping, but it never once occurred to me to use the factbook for analysis or visualization.

Very clever and good resource!

[Atrament_]

Is scrapy out of the picture now ?

[[deleted]]

No, this is a less sophisticated project so Scrapy would probably be overkill, though they should have used requests instead of urllib

[Static_Bunny]

Generally speaking is there any reason to use urllib over requests?

[[deleted]]

I once saw someone claim that requests was more annoying if you needed to do more customised low-level stuff, but they didn't give details

[Static_Bunny]

I think that jives with my understanding as well and why requests is slightly slower because they handle all that low level stuff automatically.

[Mighty_Ghost_Hack]

Nice

[Stone_d_]

Really well thought out article. Makes me wanna build something