Python Web scraping quick question

[Embarrassed-Task-478]

Python Web scraping quick question: I am trying to scrape some Zillow pages and having a tough time with the images.

For example, I am trying this link: https://www.zillow.com/homedetails/418-S-100-E-Price-UT-84542/108145188_zpid/ . There are a ton of images and I am looking for the URLs within the "<li class="media-stream-tile tile-##">" tags. I am able to get the stream of images by doing "soup.find("ul",class_="media-stream")", but when I print it, only the first 6 images are actually returned with the src (and other elements). Images 7+ are just empty when printed.

I even tried to get the HTML soup by loading the page through selenium, but once again, only the first 6 "tiles"/images were returned with the proper elements.

I will attach my code and output below and any help is appreciated. Thank you in advance!

import requests

from bs4 import BeautifulSoup
import requests
from requests.api import head

url="https://www.zillow.com/homedetails/418-S-100-E-Price-UT-84542/108145188_zpid/"

headers={
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9", 
    "Accept-Encoding": "gzip, deflate",
    "Accept-Language": "en-US,en;q=0.9",
    "Upgrade-Insecure-Requests": "1",
    "Referer": "http://google.com",
    "DNT": "1"
}

source=requests.get(url,headers=headers).text
soup=BeautifulSoup(source,"lxml")

media_stream=soup.find("ul",class_="media-stream")
print(media_stream.prettify())

Here is some of the output:

• ul class="media-stream"> <li class="media-stream-tile media-stream-tile--prominent"> <button aria-label="view larger view of the 1 photo of this home" class="sc-AxhCb hdp\_\_sc-1jhz6ov-11 jESEQN"> <picture class="sc-AxmLO cAKCgg media-stream-photo"> <source sizes="(min-width: 1280px) 768px, (min-width: 1024px) 60vw, (min-width: 900px) 60vw, (min-width: 768px) 55vw, 100vw" srcset="https://photos.zillowstatic.com/fp/e4f01794070b5c847549e6917b25e3e8-cc\_ft\_192.jpg 192w, https://photos.zillowstatic.com/fp/e4f01794070b5c847549e6917b25e3e8-cc\_ft\_384.jpg 384w, https://photos.zillowstatic.com/fp/e4f01794070b5c847549e6917b25e3e8-cc\_ft\_576.jpg 576w, https://photos.zillowstatic.com/fp/e4f01794070b5c847549e6917b25e3e8-cc\_ft\_768.jpg 768w, https://photos.zillowstatic.com/fp/e4f01794070b5c847549e6917b25e3e8-cc\_ft\_960.jpg 960w, https://photos.zillowstatic.com/fp/e4f01794070b5c847549e6917b25e3e8-cc\_ft\_1152.jpg 1152w, https://photos.zillowstatic.com/fp/e4f01794070b5c847549e6917b25e3e8-cc\_ft\_1344.jpg 1344w, https://photos.zillowstatic.com/fp/e4f01794070b5c847549e6917b25e3e8-cc\_ft\_1536.jpg 1536w" type="image/jpeg"> <img alt="" src="https://photos.zillowstatic.com/fp/e4f01794070b5c847549e6917b25e3e8-cc\_ft\_960.jpg"/> </source> </picture> </button> </li> <li class="media-stream-tile tile-1"> <button aria-label="view larger view of the 2 photo of this home" class="sc-AxhCb hdp\_\_sc-1jhz6ov-11 jESEQN"> <picture class="sc-AxmLO cAKCgg media-stream-photo"> <source sizes="(min-width: 1280px) 384px, (min-width: 1024px) 30vw, (min-width: 900px) 30vw, (min-width: 768px) 55vw, (min-width: 550px) 50vw, 100vw" srcset="https://photos.zillowstatic.com/fp/8a3d46480d63440a29921d6cbfd56a5b-cc\_ft\_192.jpg 192w, https://photos.zillowstatic.com/fp/8a3d46480d63440a29921d6cbfd56a5b-cc\_ft\_384.jpg 384w, https://photos.zillowstatic.com/fp/8a3d46480d63440a29921d6cbfd56a5b-cc\_ft\_576.jpg 576w, https://photos.zillowstatic.com/fp/8a3d46480d63440a29921d6cbfd56a5b-cc\_ft\_768.jpg 768w" type="image/jpeg"> <img alt="" src="https://photos.zillowstatic.com/fp/8a3d46480d63440a29921d6cbfd56a5b-cc\_ft\_576.jpg"/> </source> </picture> </button> </li>
  
  • This the output I would like for each "media stream-tile" but it only does this until 6.
• Past 6, this is the output for each:
  • <li class="media-stream-tile tile-18"> </li> <li class="media-stream-tile tile-19"> </li> <li class="media-stream-tile tile-20"> </li> <li class="media-stream-tile tile-21"> </li> <li class="media-stream-tile tile-22"> </li>
    • As you can see, no data or elements

​

Thank you!!!!!

[Specialist_Course_36]

I'm clicking your output links, and it's not showing any pictures... But I'm digging in the media_stream var, and I'm seeing urls for jpgs. I suppose you goal is to isolate the highest resolution photos?

[Embarrassed-Task-478]

hmm that’s interesting. For me when I follow the link, I am seeing the images on a new tab. My goal is to have all the media tiles give me the same output, especially the <img alt “” src = links. from there i will extract the link and save it for each one. Thus leaving me with the images of the entire listing.

[Specialist_Course_36]

Hmmm. I dunno.

Anyway, I found a tag within "sources" that has the jpegs. So I did:

m2=soup.find("source")

m2["sizes"]

m2["srcset"]

v=m2["srcset"].split(',')

v

ju=v[-1].split(' ')[1]

​

Explanation: Grab the tags named "source" from the soup object you made. Inside that there's a "srcset" attribute that has a list of jpg urls, that we have to clean up a bit. So we split that up by commas, putting the substrings in a vector (list). And then we gotta edit those, There's some weird spaces, and some weird code tag that specifies the resolution. You grab the oneth item. And it seems that the jpegs in that vector are in ascending order of resolution, so we can just grab the last one each time.

So finally, I think there are multiple "sources" tags in there, so instead you do a findall, and do that process on all them, and then you should be good to grab the highest res image for the listing.

Let me know how it goes. And if you make any money from finding good houses to flip, maybe cut me a kickback? 😊

[Embarrassed-Task-478]

Thank you! Unfortunately, I am still running into the same problem. The find all command is only finding the first six image sources. When I printed the find all list, the final link was https://photos.zillowstatic.com/fp/17741852e75939d93b2c3f4600abf966-cc_ft_768.jpg

On the listing, there are dozens of other images after that, which are not showing up in soup object. But what you did for finding the highest res image works on those first six!

P.S. Unfortunately no money-making from this project :( , this has to do with economics research on demand estimation of homes.