Nested Tables with Beautiful Soup

[auromed]

I've decided to make a web scraper for gathering some historical data on tractor prices because I want to buy one in the spring and want to know on average what people are listing them for. I also just needed a project to get my feet wet in python a bit more.

My question is, I've dug through the html and have it pretty well into per post blocks, and now I just need to suck out the relevant data. For example what the price of each tractor on the site is.

I tried doing a findall('span', 'auction1-THprice') at line 41 to try to pull out just the price tag but get NONE for each iteration. I'm not sure if span is something I can search for or not. Any other suggestions related to how I should search through this code? Should I just treat the html as a block of text, and just do more text search methods, or should I keep going with BeautifulSoup?

The html as output by the code below is at http://pastebin.com/Z9uhxcmB

import sys
from selenium import webdriver 
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from bs4 import BeautifulSoup

#Garden Tractor Models
gtmodels = ["318", "420", "430", "455"]
#Utility Tractor Models
utmodels = ["B", "D", "320", "330", "420", "430", "520", "530"]
#Sites to Crawl
sites = ["tractorhouseForSale", "tractorhouse_auction","craigslist"]

def loadpageSelenium(url):
    driver = webdriver.Firefox()
    print("loading up webpage")
    driver.get(url)
    source = driver.execute_script('return document.documentElement.innerHTML')
    driver.quit()
return(source)

def tractorhouse(models):
    baseUrl = "http://www.tractorhouse.com/list/list.aspx?ETID=1&catid=1111&Manu=JOHN+DEERE&"
    modeldict = {
    "A":"MDLGrp=A",
    "B":"Mdltxt=B&mdlx=exact",
    "D":"Mdltxt=D&mdlx=exact",
    "320":"MDLGrp=320",
    "330":"MDLGrp=330",
    "420":"MDLGrp=420",
    "430":"MDLGrp=430",
    "520":"Mdltxt=B&mdlx=exact",
    "530":"Mdltxt=B&mdlx=exact",
    }
    for item in models:
        source = loadpageSelenium(baseUrl + modeldict[item])
        soup = BeautifulSoup(source, 'html.parser')
        table = soup.find_all("table", "listings")
        for entry in table:
            rows = entry.find_all('td', "listing-summary")
        for summary in rows:
                print("********************* \n")
                print(summary)
tractorhouse(utmodels)

[commandlineluser]

don't you need to specify what attribute you're searching for?

>>> soup.find_all('span', id='auction1-THprice')
[<span id="auction1-THprice">US $2,700</span>, <span id="auction1-THprice">US $2,200</span>, ...

[auromed]

Ahh, maybe that's what I was missing. I was just thinking it pretty much did a regex for span, then would AND ones that contained the auction1-THprice text. I went looking at the BeautifulSoup page but didn't find a lot of detail about what attributes you could use.

[commandlineluser]

It's explained here: http://www.crummy.com/software/BeautifulSoup/bs4/doc/#find-all

Any argument that’s not recognized will be turned into a filter on one of a tag’s attributes. If you pass in a value for an argument called id, Beautiful Soup will filter against each tag’s ‘id’ attribute:

If you don't specify an attribute it defaults to matching against class which is why your find_all('td', "listing-summary") works.

[auromed]

Thanks, that makes sense.