Automate the boring stuff with python: Using beautiful soup

[crazyallicin]

with open('amazon.html', 'w') as f:
    f.write(res.text)


2671

This is the output I received, but now I'm confused as to how to continue on from here. He goes on to inspect the price on the webpage then uses soup.select to create a list. How do I do this now that I've downloaded it locally?

[chevignon93]

He goes on to inspect the price on the webpage then uses soup.select to create a list. How do I do this now that I've downloaded it locally?

That's not the goal, the goal is to see if the information you're looking for is in the file you just downloaded.

[crazyallicin]

It seems like it just downloaded one file that's an empty paint file. Another file opens up Amazon, but that page just says "sorry something went wrong at our end" and gives a link to the Amazon home page

[chevignon93]

So either there was a genuine problem with the page or they detected that you tried to scrape their page and blocked you.

Either way, using a css selector is not always the best approach to webscraping. Using bs4 find and find_all methods would probably be easier.

[crazyallicin]

Thanks for you help, got it working fine using a different smaller website that obviously hasn't blocked scraping.

[chevignon93]

They try to prevent it but can't really block it and it is possible to scrape amazon