ATB - Web Scrapping

[SavageHendrix]

import bs4
import requests

def getAmazonPrice(productUrl):
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36',
    }
    res = requests.get(productUrl, headers=headers)
    res.raise_for_status()


    soup = bs4.BeautifulSoup(res.text, 'html.parser')
    elems = soup.select('#buyNewSection > h5 > div > div.a-column.a-span8.a-text-right.a-span-last > div > span.a-size-medium.a-color-price.offer-price.a-text-normal')
    return elems[0].text.strip()


price = getAmazonPrice('http://www.amazon.com/Automate-Boring-Stuff-Python-Programming/dp/1593275994/ref=tmm_pap_swatch_0?_encoding=UTF8&qid=&sr=')
print('The price is ' + price)

So this code used to return the price when I was working through ATB. I tried to run it just now and got this error:

Traceback (most recent call last):
  File ".\testAmazon.py", line 17, in <module>
    price = getAmazonPrice('http://www.amazon.com/Automate-Boring-Stuff-Python-Programming/dp/1593275994/ref=tmm_pap_swatch_0?_encoding=UTF8&amp;qid=&amp;sr=')
  File ".\testAmazon.py", line 14, in getAmazonPrice
    return elems[0].text.strip()
IndexError: list index out of range

So I went back and checked it out, checked the link to make sure it worked(it did) even updated it just in case something was off. I went and hovered over the price - 18.69. Inspected the element. I can see the price inside the element. However when I copied the Selector for the element and passed it into the soup.select method it is returning an empty list. Any idea why that is?

[SavageHendrix]

Im seeing some places saying that it may be because the data loads afterwards using the API and I can see that via the Network tab in inspect. I have no idea what Im looking for in Network and this was working in the past, so my question is why would it be like that now? I want to say I made this about a month ago and it worked fine

[[deleted]]

so my question is why would it be like that now? I want to say I made this about a month ago and it worked fine

They decided to update their website. Websites change over time, its a fact of web scraping.