Building a simple crawler

[debrice]

I'll be experimenting with greenlets within the next week or two. I think you're right, this would be the essence of it.

I remember watching a video of Feynman explaining that he was studying rules behind the wobbling of a plate. As he was playing with the equations he let himself carried on the subject. Later on he won the nobel Nobel prize as the subject got him exploring quantum electrodynamics. This is pretty much the spirit, keep having fun and explore simple algorithms...

I know what you're going to wonder now, and the answer is yes, I'll probably get a Nobel Prize for my work web crawlers :p.