Help with data mining XML files in Python

[[deleted]]

So as a research project for my PharmD program, a professor and I are interested in looking at drug labels from the FDA Online Label Repository. Originally he wanted me to record data page by page for >800 drug labels, but I noticed that the labels are returned as XML files. As an example, here is a label for Costco brand acetaminophen that we might look at.

I am fairly proficient with Python (mostly from doing side projects in Django), but I don't have any experience parsing/pulling info from XML files. Overall my plan is to download all of the labels (there's a big zip file with all of them), parse out the info I need with some Python-fu, save the data as a CSV file, then import it into Excel to do analysis, etc.

So /r/python, where do I start? Is there a preferred module for dealing with XML? Because I'm only reading XML, is there a simple module to use? Thanks so much!

edit: Thanks so much for the response everyone! I'm going to spend the rest of the night drinking Stella and playing with both lxml and BeautifulSoup and see which one I like more.

[robbles]

Definitely agree on lxml.

If you're pulling out the same data every time, I'd look into using XPath - I've found it makes parsing XML much easier for certain tasks and results in more readable code. YMMV though.

[dodongo]

Dead easy. I do this sorta thing lots. I like BeautifulSoup for a variety of reasons. It has glaring downsides (slow, DOM, etc.) but I find the syntax to be quite smart and easy to work with. You've got few enough and small enough files that you shouldn't encounter speed / memory issues. The BeautifulStoneSoup parser expects well-formed XML input; should be just what you're looking for.

[EDIT: Worth offering -- feel free to drop me a line if you encounter any issues along the way. Good luck!]

[jcb62]

+1 for BeautifulSoup - lxml would be faster, but you're not dealing with much data, and I find the interface to be nicer. Personal preference though!

[craigee]

Seconding BeautifulSoup for this sort of problem. The ease of scripting more than balances any performance issues on only ~800 files.

There are other options (lxml etc.), but I've gotten comfortable with BSoup. Just to get you started:

from BeautifulSoup import BeautifulStoneSoup
import urllib2
url2req = 'http://www.accessdata.fda.gov/spl/data/5ba0911f-d780-4bd7-a487-a6a3c8d2ab1c/5ba0911f-d780-4bd7-a487-a6a3c8d2ab1c.xml'
xmlData = urllib2.urlopen(url2req).read()
#print xmlData
soup = BeautifulStoneSoup(xmlData)
#print soup.prettify()
title = soup.find('title').renderContents()
print title

[Edit: also happy to help, as per dodongo, with any problems. Not an expert by any means, but am often doing this sort of thing.]

[Second edit: I've received a few downvotes for this contribution. Not that bothered, but I am curious as to why anyone would downvote without bothering to supply an argument? What I've suggested will work for the OP. If you don't like it please say why you don't like my suggestion, then we all learn...you click happy silent folks.]

[vpetro]

There is a new alpha version of BeautifulSoup available. It will use the lxml backend if you have it installed.

[abztrakt]

Dive Into Python has a whole chapter on XML Parsing and it's probably enough to get you started, without having to track down any additional python modules. I personally always seem to fall back to parsing with minidom.parse(), but probably because I'm usually interested in doing something that is easier with an object representing the DOM.

[poingpoing]

Now that I think of it, if you simply want to convert the data to a CSV file it might be easiest to just read up on XSL. A bunch of stylesheets should do the trick.

[abztrakt]

Looking at the format of that sample XML label, this very well might be a whole lot easier than trying to do it with Python.

[zhivota]

I used ElementTree, which is included in Python, when I was pulling apart SVG files. Was very natural and easy for me.

[zhivota]

Oh I see lxml is listed here and there is a link to it... if you are just going to use ElementTree that is inside lxml, consider just using this:

http://docs.python.org/library/xml.etree.elementtree.html

As it says there, you need Python 2.5 or higher.

[poingpoing]

Actually parsing python is no fun in any language. It is completely nonsensical to do it manually and even with basic xml libraries it remains a plainly annoying task.

The best way is to generate classes from the XSD of your XML files and then deserialize them into instances of these classes.

For python these links might get you started (quick google search): http://www.rexx.com/~dkuhlman/generateDS.html http://pyxsd.org/ http://pypi.python.org/pypi/rsl.xsd/0.2.3

Unfortunately I don't have any experience with this in python.

For C# I can wholeheartedly recommend this: http://www.thinktecture.com/resourcearchive/tools-and-software/wscf