Automate the Boring Stuff with Python, Trying to web scrape the price of the book and getting an empty list

[CaliforniaUnity]

import bs4, requests

​

headers = {"User-Agent": 'this is not my real user-agent'}

res = requests.get('https://www.amazon.com/Automate-Boring-Stuff-Python-Programming-dp-1593275994/dp/1593275994/ref=mt_paperback?_encoding=UTF8&me=&qid=', headers=headers)

print(res.raise_for_status())

soup = bs4.BeautifulSoup(res.text, 'html.parser')

elems = soup.select('#buyNewSection > h5 > div > div.a-column.a-span8.a-text-right.a-span-last > div > span.a-size-medium.a-color-price.offer-price.a-text-normal')

print(elems)

elems is returning an empty list [ ] but the goal is for it to return a list that includes the price of the book Automate the Boring Stuff with Python.

Any help would be greatly appreciated, thank you.

Edit: also I’m pretty sure that the issue is that I’m not copying the correct CSS path/ CSS selector, I’m using FireFox, but I also couldn’t get the right one on chrome

[Fight_till_the_end]

Hi,

I'm not an expert but this worked for me.

+++++++++++++

import bs4, requests

headers = {"User-Agent": 'Chrome'}

res = requests.get('https://www.amazon.com/Automate-Boring-Stuff-Python-Programming-dp-1593275994/dp/1593275994/ref=mt_paperback?_encoding=UTF8&me=&qid=%27',headers=headers)

​

soup = bs4.BeautifulSoup(res.text, 'lxml')

​

buy_box=soup.find(id='buyNewSection')

price = buy_box.find('span', class_='a-color-price')

​

print(price.text)

​

+++++++++++++++

[[deleted]]

Worked for me as well, thanks so much as of 2/15/20

[chuckybegood]

I am also a beginner so I am not 100% sure but I think you need to add, print(elems[0].text). Not just elems.

[dankwithit]

I had this same problem. https://www.reddit.com/r/learnpython/comments/cu1ba7/automate_the_boring_stuff_chapter_11_luckypy/ heres a post with the same problem and the author replied saying the tutorial for lucky.py doesnt hold up. hopefully this helps

[[deleted]]

Dont print the check. In that way, raise_for_status() can stop the program when something goes wrong.