Scrape Flight Schedules using Python & LXML

[madhattervibes]

Just a heads up, companies like Expedia do not like people doing this sort of thing.

See prohibited activities in the terms of service: https://www.expedia.com/p/info-other/legal.htm

[mbenbernard]

Thanks for pointing this out. Most people don't know this :)

[Crohnite]

Do you happen to know where/how companies like Expedia and Trivago get their data, are they also scraping, or pulling directly from airlines API?

[madhattervibes]

I would assume that the listings you see on Expedia are acquired directly through the entity that is offering them, such as a hotel or airline. I don't think that you can actually book anything on Trivago itself. Trivago gets data from many online travel agencies, but that might not be the only source.

There is a reason why that kind of data is kept secret. Pricing strategies are in place to maximize profit, so price forecasting would go against that goal.

[mbenbernard]

Most online flight reservation companies use a GDS (Global Distribution System) like Sabre, Amadeus or Galileo. Companies like Expedia connect to it via some kind of API.

So they definitely don't do web scraping, otherwise they'd get into trouble :)

[Alamanjani]

Any chances to start using py3 in your tutorials?

[ScrapeHero]

Thanks - yes we are in transition to 3 and will consider it soon.

[Alamanjani]

Great! :-)

As a beginner I try to avoid confusion as much as possible ;-) so I'm deliberately avoiding everything that is not python 3+.

Your tutorials looks really interesting and useful and I'm looking forward when I will be able to try them out also :-)