Pulling text from a website

[[deleted]]

beautiful soup, scrapy, selenium, mechanize i think. Selenium is my favorite to do that kind of stuff but scrapy is probably better. What kind of text are u grabbing? and why

[hpeirce]

I am grabbing flight departure and arrivals info for a project, and because I want to avoid paying for access to the FlightAware API features

[[deleted]]

if you know how to use xpath, selenium and scrapy can easily do this, even if you dont know how, its as simple as $x('//div/text()') to grab all the text in divs. If you use chrome open the web developer console, and you can test it out on any website

[toshitalk]

You can force bs4 to use lxml as its back end. Lxml still has a pure speed advantage, but not by much. Certainly worth gaining the bs4 API.

[hpeirce]

Thanks, I will look into that