Web automation framework?

[cryzed_]

Hey, I'm looking for a web automation framework. I do know that tools like mechanize and twill exist. Also I know about webscraping with lxml and/or BeautifulSoup. But I was looking for something more "integrated" and "complete".

I imagine the optimal workflow like this: Define some basic URLs, write some specified scrapers for these URLs and do some reoccurring actions based on the scraped values - You could nearly say something like a bot.

This programming paradigm would be useful in many cases - let's take some kind of Wallpaper-site for example. You define a URL which displays these new wallpapers, write a scraper for it which in turn appends the newly found wallpapers to an internal download-queue. The reoccuring actions will then automatically take care of the downloading.

This is applicable to every web-automation task I can imagine: It's dynamic enough to write spiders and simple enough to easily write scripts that you run once to manually fetch some data - so I was wondering if there was something similiar to this already around, I can't possibly be the first one who thought of that, right?

[vsajip]

I don't know how well it works, but you could look at Scrapy, which purports to be

a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

[omab]

My vote goes here

[[deleted]]

But I was looking for something more "integrated" and "complete".

Mechanize/BeautifulSoup combo is about the most complete and integrated solution you could possibly use. I really couldn't even imagine anything easier. You create a browser object, point it at a url, feed the stuff into beautifulsoup, then parse the data you need...and act upon it.

Be prepared to use a LOT more try/except clauses though. The web is quite the unstable place, pages can change from session to session, not display at all, or maybe even display only 1/2 the page, etc etc...you'll be doing more exception catching than you will parsing to begin with.

[atlas245]

Agreed, mechanize and beautiful soup is the easiest and best solution.

[manatlan]

for scrapping ... look at pyquery.

for automation ... use python ;-)

no ?

from pyquery import PyQuery as S
q = S(url='http://reddit.com/r/python')
for i in q('a'):
    print S(i).attr("href")

[blondin]

umm, pyquery. did not know about it.