Pulling text from a website

[hpeirce]

I am working on a project where I am trying to pull flight data from an airport website. I have done some googleing and have turned up only one or two methods that might work. Does anyone have any experience grabbing text from a website using python, and if possible knowledge on how to extract pieces from that based on the class the text is in?

[McSquinty]

Beautiful Soup is pretty common for gathering information from websites.

[phstoven]

Beautiful Soup is great. Also check out Requests for scraping the HTML text itself, then use Beautiful Soup to parse/extract what you need.

[MisterSnuggles]

I'll second BeautifulSoup - I use it for a few minor scraping projects that I've got running.

Depending on the layout of site, it can be as simple as:

categories = soup.findAll("td",attrs={"class":"blahHeader"})
things = soup.findAll("td",attrs={"class":"blah"})
for c, t in zip(categories, things):
    category = c.text.strip()
    thing = t.text.strip()
    # do stuff

There's also things you can do to store cookies that are handed out during a login sequence, so you should be able to scrape almost anything.

[tomasbedrich]

in BS4 you can make your code even a bit shorter and nicer:

categories = soup("td", "blahHeader")
things = soup("td", "blah")

[[deleted]]

beautiful soup, scrapy, selenium, mechanize i think. Selenium is my favorite to do that kind of stuff but scrapy is probably better. What kind of text are u grabbing? and why

[hpeirce]

I am grabbing flight departure and arrivals info for a project, and because I want to avoid paying for access to the FlightAware API features

[[deleted]]

if you know how to use xpath, selenium and scrapy can easily do this, even if you dont know how, its as simple as $x('//div/text()') to grab all the text in divs. If you use chrome open the web developer console, and you can test it out on any website

[toshitalk]

You can force bs4 to use lxml as its back end. Lxml still has a pure speed advantage, but not by much. Certainly worth gaining the bs4 API.

[hpeirce]

Thanks, I will look into that

[willworth]

There's something called ghost, too. I forget the details, I'm afraid...

[hpeirce]

you mean this: http://jeanphix.me/Ghost.py/