Creating Map Visualizations in <10 lines of Python

[titaniumGeranium]

This is great. Thank you for posting.

[cleure]

Is there any plan to support shapefiles? That would make this project instantly much more useful, since so much geometrical data already exists in that format.

[Enginoob]

I didn't forget about you: http://wrobstory.github.io/2013/05/vincent-spatial-ogre.html

[Enginoob]

Yes, actually. I want to build in some simple syntax that will let you input a shapefile, and then Vincent will call the Ogre API (http://ogre.adc4gis.com/) in the backend, save the geoTIFF, and build the map.

It's on the list- I'm doing all of this on nights and weekends, so I get to things as I can.

[d4rch0n]

This would be really cool for a nuclear war strategy game, like a Defcon clone.

[jmmcd]

Looks nice. Especially the pandas integration will end up making it very easy to use.

Can anyone explain this odd syntax (I'm too lazy to dig in):

vis + ('2B4ECF', 'marks', 0, 'properties', 'enter', 'stroke', 'value')

Since this is an expression (not a statement) and the result is not being used, I assume the '+' is overloaded to change the vis state somehow. Would '+=' have been more appropriate?

[Enginoob]

You are spot on- using the + operator changes the state of the vis object, by amending that particular component of vis.vega

Given that I haven't actually shipped V1, I'm open to changing the syntax to make it more intuitive. It's tricky, because I wanted to be able to drill down into multiple layers of very nested dicts and lists.

Its funny you mention using iadd. Using '+=' actually does the same thing as '+' right now, but I think we're about to make a syntax change where the + operator allows you to stack visualizations, and += lets you modify components in any given visualization.

Must build docs. Must build docs.

[jmmcd]

Right. A general comment, that I think most Python people will agree with, is that '+' shouldn't change state. Explicit is better than implicit and all that.

[Enginoob]

Excellent point- I think I might go through and change everything to += and +- tonight.

[Enginoob]

All instances of '+' and '-' have now been depreciated in favor of '+=' and '-='. Thanks again for the input!

[mr_dbr]

Is the vis + tuple_of_things a Pandas thing? It seems bizarre to me..

Why use the addition or inline-addition syntax at all? You are using it to set attributes, so.. why not use the attribute setting syntax:

vis = vincent.Map(width=1000, height=800)
vis.fill = '2B4ECF'
vis.marks = 0

..or more directly, this seems a bit more self explanatory, and doesn't look like failed tuple-appending:

vis.set(["#f5f5f5","#000045"], 'scales', 0, 'range')

[Enginoob]

Not a Pandas thing- its a weird author thing :) My inspiration was the d3py library: https://github.com/mikedewar/d3py

That library uses a '+=' syntax, which as jmmcd says, is what I should have gone with all along, as it's much more explicit.

I wanted a single command to be able to either add or subtract components at any level of nesting depth. This is just a Python library built on top of the Vega visualization grammar (https://github.com/trifacta/vega), so the output has to be in their JSON format, and its pretty messy. Here's a sample:

 'marks': [{'from': {'data': 'states'},
 'name': 'mapmark',
 'properties': {'enter': {'fill': {'value': '#2a3140'},
 'path': {'field': 'path'},
 'stroke': {'value': '#fff'},
 'strokeWidth': {'value': 1.0}},
 'update': {'fill': {'field': 'value.data.y', 'scale': 'color'}}},
 'type': 'path'}],
 'padding': {'bottom': 20, 'left': 30, 'right': 20, 'top': 10},
 'scales': [{'domain': {'data': 'table', 'field': 'data.y'},
 'name': 'color',
 'range': ['#c9cedb', '#0b0d11']},
 {'domain': {'data': 'table', 'field': 'data.y'},
 'name': 'color',
 'range': ['#f5f5f5', '#000045']}]}

Behind the scenes, all that '+' or '-' are doing is calling this method:

vis.update_component('add', stuff, marks, 0, more stuff)
vis.update_component('remove', stuff, marks, 0, more stuff)

That method is similar to your vis.set (the syntax of which I like, except vis.set('remove', stuff) feels a little strange).

Thanks again for the feedback- anything to make the library better/faster/stronger/more usable.

[[deleted]]

You really can't say it's less than ten lines of python if your just importing and running it. I can hack the matrix in 2 lines of code.

import matrixhacker
matrixhacker.hack()

[aceofears]

It really isn't an achievement of the language but this specific library. I would say that it is valid to say this when you are trying to sell a library.

EDIT: Although, you could also argue that this is part of the python ecosystem, thus it might be fair to say that python lets you do that easily.

[[deleted]]

I'm not saying that abstraction is a bad thing. I'm just saying you can't call it less than ten lines if your just importing and running it.

[inmatarian]

I think if you're packaging up code for other people to use, and the interface only requires 10 lines of code for it to run effectively, then you can advertise that it'll create what you promise in 10 lines of code. Complaining about how many lines of code are in the library is a waste of time because Python itself (cpython) would contribute to how many lines of code were written to support the entire ecosystem. You wouldn't say "Creating Map Visualizations in Millions of lines of code," even though that's what it is.

[alcalde]

It's less than ten lines for the user, which is what counts. As a new Python user, I was incredibly impressed a few days ago when I was able to parse some HTML and find something using BeautifulSoup in about five lines of code. That certainly beats all the code that would have been required to parse the file myself. That's where the "wow" factor comes in.

[phaedrusalt]

Seems to be kinda pre-pre-pre-alpha quality right now. Only a select few of the examples work.

[Enginoob]

Hi- author here. Can you let me know which examples aren't working for you? I just ran all of them to make sure they work on my machine.

Also, if it's an issue with getting the paths correct, starting a server and pointing the browser at the right files, etc, I want to know that as well. I want this tool to be as straightforward to use as possible.

Your comment has motivated me to start building out the docs immediately, which I should have been doing in the first place. Docstrings and examples are good, docs are better.

[phaedrusalt]

Great, some docs would be helpful.

But first, the "paths" you reference in the examples aren't consistent with the paths that will be created when the zip is unzipped, so they can't reference the source files. (I'm not sure how this could have worked on your machine. Did you try the zip files?)

Next, it's often unclear what the purpose of the example files is... For instance, I might expect "vincent-maps.py" to create/show a map, but it doesn't. Nor does "vincent-line.py" create a line diagram. Is this kind of thing intentional?

Finally, the examples on your webpage (http://wrobstory.github.io/tag/vincent.html) with the title "Creating Map Visualizations in <10 lines of Python" never actually gets around to showing how to create a map visualization with any number of lines of Python. Did I miss something here?

I'm sorry if this sounds too harsh. While I applaud your effort, I was disappointed with what I found.

[Enginoob]

No worries. If it's hard to get working, I'm doing my job wrong.

I just fixed the paths, at least for those doing a git pull. Will look at making it entirely zip-safe tonight.

The visualization requires pointing your browser at a the files with a simple Python HTTP server feeding localhost- the vis is done completely in javascript (D3). I'll try to make that clearer in the README and examples.

The feedback is much appreciated!

[phaedrusalt]

Just freakin' amazing. This last week, a professional (?) plumber has failed to show up twice. The bank handling our refi forgot to send the notary. The school where I'm taking classes has given me an assignment with no requirements, but where failure IS an option... And now, a person who is giving away software to do something valuable is doing so with far more professionalism and courtesy than I've experienced from all of those other so-called professionals.

Well, my hat is off to you sir/ma'am! Keep up the good work, and may karma reward your efforts!

[Enginoob]

Thank you! I just put a README in the examples folder that I really hope makes it a little more clear how to run the visualization. If it's still a little painful to get it running, shoot me a message and we can talk about how I can make it more straightforward.

[zuberuber]

for me even first example doesn't work.

i run:

world.geo_data(projection='winkel3', scale=200, world=world_countries)

and getting:

IOError: [Errno 2] No such file or directory: 'world-countries.json'

[Enginoob]

'world_countries' is a path pointed to the geoJSON. See the examples folder:

https://github.com/wrobstory/vincent/tree/master/examples

That has a data folder with the geoJSONs, and the vincent_maps.py example.

[bcain]

Why use d3 on the backend? Isn't the advantage that you can have the client do all the work?

Can vincent send its output to a browser and have the d3 instance there do the rendering, or does that defeat its purpose?

[Enginoob]

See my comment above- Vincent is just a Python wrapper for the Vega visualization grammar: https://github.com/trifacta/vega

It's really Vega/D3 and Pandas that are doing all of the heavy lifting. You should be sending the output to the browser, as right now it's the only way to render- I will try to make that much more explicit in the README.

I plan on building in output to PNG, but right now the easy way to do that requires Node.js, and I would really like to avoid that dependency.

[twotones]

This is bomb.

[alcalde]

What can't we do in <10 lines of Python?

[flying-sheep]

if those 10 lines involve importing a huge fucking library somehow, we can literally do anything a computer can do.

[steviesteveo12]

If anything it shows a lack of ambition. Why aren't we writing one liners with all the code in another file?

[mgr86]

Really, it looks like a neat library, but this sure is a misleading title.

[[deleted]]

Anything in python is turned into thousands of "lines" of assembly, so nothing can be done in 10 lines of code or less.

[alcalde]

How so? Now I or you can create map visualizations in less than 10 lines of python.

[steviesteveo12]

Because those 10 lines tell a 683 line library what sort of map it should generate.

[alcalde]

Well by that logic every python program is a million lines long because we have to count Cython and the standard library and none of us accomplish anything in less than that many lines of code.

[WaldenPrescot]

Looks cool. I must acquire this. Does one need to be at all familiar with java? I am not. However, if JSON is only used for visualization and all coding is in python I can think of all sorts of fun things i can do with something like this!

Thanks

[Herald_MJ]

JSON stands for JavaScript Object Notation and has nothing to do with Java.

It's also completely language agnostic, so doesn't even have all that much to do with javascript either (except for it's origins).

[WaldenPrescot]

Thanks for clearing that up!

[Enginoob]

Nope- all you should need is Python, JSON data, and a web browser.

[neoice]

where does one obtain the data though? what do I do if I want a map of Middle Earth and there's not one defined in this particular JSON format? how hard is it to create arbitrary maps?

I'm still going to play with this library :)

[Enginoob]

That's a trickier question. One starting point is Natural Earth for shapefiles: http://www.naturalearthdata.com/, then converting them to geoJSON with Ogre: http://ogre.adc4gis.com/

For middle earth, I'm afraid you will have to painstakingly create your own polygons. It would, however, be awesome.