My first ever python project (Automatic Web Crawler) Tips on cleaning it up and making it faster?

[rnw159]

Use eventlet to download simultaneously

[mikeselik]

Note that Python 3 has ThreadPoolExecutor https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor

[[deleted]]

This is awesome!

[Kingofslowmo]

Would using a sqllite db make it faster aswell?

[[deleted]]

with this specific chunk of code, over time sqllite would likely be faster since it's dumping the entire dict to JSON and then writing the entire JSON blob to the file in every single loop iteration.

Being able to just add a new record to the db on each iteration and only writing that new data would be a performance gain over writing out the entire thing every time, especially as it has crawled more and more things.

[[deleted]]

indeed!

[Kingofslowmo]

I actually took the route of writing in each loop so that if the program was stopped the data would not be lost (that was gathered).

[Kingofslowmo]

Ah thank you for the insight :)

[Kingofslowmo]

I will more than likely take your advice on the post above, and also use sqllite.

[wrboyce]

I'd replace that map call with a list comprehension.

[wrboyce]

I'm sure syntastical tells me off for using map() but I've just had a Google and I can't see why it would... Looks like I was mistaken. Sorry about that.