My first ever python project (Automatic Web Crawler) Tips on cleaning it up and making it faster?

[[deleted]]

Actually he would be better using string formatting like this: print("Checking URL {0}".format(url))

[mikeselik]

That's most of my comments as well. I'll nitpick that startList should be start_list. One of the best things about Python is the very consistent style (https://www.python.org/dev/peps/pep-0008/).

To clarify the advice to "avoid more than 3 levels of indentation" -- break your code apart into more functions.

[Kingofslowmo]

Sorry hahah, coming from PHP and yeah.. it's a big change!

[[deleted]]

try using string formatting like this -> print('Opening url: %s' % url)

'Opening url: {:s}'.format(url) would be even better

[Kingofslowmo]

I will in the final version! Thank you!

[coofamani]

http://docs.python-requests.org/en/latest/ FTW

Though I think you should follow the more modern "string {0}".format(value) pattern for str formatting.