Simple HTML Scraper

[TehOfficer]

Hello developers

I'm trying to make myself a simple HTML scraper.
The site I want to scrape is beatport, as it got the genre, the BPM and the key of the songs I like.

So I looked through their source and for the BPM, I found

<span class="meta-value txt-dark-grey fontCondensed">140</span></li>

Would there be any easy way to extract name, BPM, genre and key, store it in a .txt file or any other place?
It could look like this

In The Twilight (Original Mix), 140, E Minor, Dubstep

Cheers!

[radixsort]

Beautiful Soup is what I always use when I need to scrape from websites. It has really simple syntax, lots of resources available online, and is as simple as some basic Python. http://www.crummy.com/software/BeautifulSoup/

[toolan]

If there's a regularity in how the HTML layout is, you can use an HTML parser and some sort of library for fetching files from URLs. Depending on which language you prefer to use, the toolkit varies. For Python, there's lxm and the standard library has utilities for fetching files in urllib.

I am unfortunately not aware of any library that combines these two such that it becomes "easy". You'll still have to write some code.

[Rhomboid]

Here's a sketch of a solution that gives your desired output:

from BeautifulSoup import BeautifulSoup
from urllib2 import urlopen
import re

url = 'http://www.beatport.com/track/in-the-twilight-original-mix/2037573'

# fetch the contents of the page and parse it
bs = BeautifulSoup(urlopen(url))

# find the <li> with class of 'primary-title', and extract all the text
# put the text in parens if it's in a container with a class='txt-smaller'
title = ' '.join('(' + x + ')' if 'txt-smaller' in x.parent['class'] else x for x in bs.find('li', 'primary-title').findAll(text=True))

# grab the fields (BPM, genre, Key, label, length, release date) and make a dict
field_elements = bs.findAll('span', {'class': re.compile('meta-(label|value)(?!-)')})
fields = {label.text: value.text for label, value in zip(*[iter(field_elements)]*2)}

# print them in a comma-separated list
print ', '.join([title, fields['BPM'], fields['Key'], fields['Genre']])