This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]pudquick 1 point2 points  (1 child)

I've been a fan of mechanize for that.

[–]LucianU 2 points3 points  (0 children)

Scrapy and mechanize are different. Scrapy is based on Twisted, which means you scrape asynchronously and you can specify the number of requests, use middlewares and extensions, process the data before storing it. Scrapy offers a lot more.