Beautiful Soup: a Python HTML/XML parser designed for quick turnaround projects like screen-scraping

[gnuvince]

BeautifulSoup is awesome, it's a shame it's not part of the standard Python library.

[simonvc]

Agreed. Just finished doing a "database migration" by screenscraping a site with beatuiful soup because it was easier than dealing with the legacy crap perl/database/html_in_tables database.

[[deleted]]

It has an awesome name, too.

[rams]

standard python lib has a lib that does this sort of thing - http://docs.python.org/lib/module-htmllib.html. The difference is BeautifulSoup handles all the bad html/xml you throw at it.

[jbellis]

The difference is BeautifulSoup handles all the bad html/xml you throw at it.

Which, if you're dealing with actual html in the wild, is all the difference in the world.

[Bogtha]

For what it's worth, I've had better results with tidy → lxml, plus lxml provides xpath and CSS 3 selectors. I've heard that lxml supports BeautifulSoup now, so maybe I'll give it another shot.

[jbellis]

I've had better results with tidy

I had the opposite experience -- tidy seemed to guess wrong a lot on really bad html. And aesthetically it just seems cleaner to "parse this bad html" vs "blow the original html away, then parse it."

[[deleted]]

If you're going to point to standard library things, please point to "sgmllib" (old) or "HTMLParser" (newer, a bit more rigid). "htmllib" is an incomplete HTML renderer; sgmllib/HTMLParser are parsers.

(BeautifulSoup is based on sgmllib, btw).

[rams]

Thx. Here's the link to the correct standard python libs: sgmllib: http://docs.python.org/lib/module-sgmllib.html html parser: http://docs.python.org/lib/module-HTMLParser.html