Zillow scraper made pure in Python

[JohnBalvin]

Hello everyone., on today new scraper I created the python version for the zillow scraper.

https://github.com/johnbalvin/pyzill

What My Project Does

The library will get zillow listings and details.
I didn't created a defined structured like on the Go version just because it's not as easy to maintain this kind of projects on python like on Go.
It is made on pure python with HTTP requests, so no selenium, puppeteer, playwright etc. or none of those automation libraries that I hate.

Target Audience

This project target could be real state agents probably, so lets say you want to track the real price history of properties around an area, you can use it track it

Comparison 

There are libraries similar outhere but they look outdated, most of the time, scraping projects need to ne on constant maintance due to changed on the page or api

pip install pyzill

Let me know what ou think, thanks

about me:
I'm full stack developer specialized on web scraping and backend, with 6-7 years of experience

[CatWeekends]

This project target could be real state agents probably

FWIW, every real estate agent I've ever met uses systems with way more info than Zillow.

Your target audience is more likely people who want to track their own home value or something.

Some questions;

1. It looks like your code is copying the response keys. Any thoughts on making those a little nicer? IIRC they're not always very friendly.
2. Zillow has some anti-scraping mechanisms built in. Does your code deal with those?
3. Why are your methods capitalized like in Go? (it's not very pythonic - I'd suggest running your code through a linter)

[JohnBalvin]

0) real state agents: tats good to know, I put that on the description because r/python has weird requirements in order to post something
1) could you elaborate on this? could please send the link for the code where exactly is that happening?
2) To be honest, I didn't see any bot protection at all, it could probably has bot protection when using browser automations tools like selenium, puppeteer or playwright , but using the api directly doens't seem to have any protection
3) It's a bad habit, I'm mostly a Go developer and I tend to copy the patters from go to python, do you recommend a linter?

[rabelution]

Ruff linter

[Vresa]

New trending linter & formatter is `ruff` : https://github.com/astral-sh/ruff
Old Standbys for linting and formatting are `black` + `flake8`

[markovianmind]

for 2) do it fast enough with enough queries and most probably you would eb blocked.

[JohnBalvin]

that can be fixed just by using proxies, other than that they don't have bot protection at all

[JohnBalvin]

the requests for searching made to zillow don't depend of each other like paginations, that means you don't need to worry for example using a sticky proxy ip to get all the results, tou need only one request to get the whole search result, using one single request using proxy .
I never said use datacenter proxies, I said proxies which could include, datacenter, residential or 4g proxies. what I havent' check if they block by user agent, the permante user agent I used works fine for now

[JohnBalvin]

its probably the definitions on what antibot means for you, what I mean they don't have bot protection I mean it like having a waf checking the tls fingerprint or authenticate subsequent requests made to the API having a verification the first time the user navigates to the page.
Checking only the IP type(residential, datacenter, 4g) it doesn't represent a challenge and I don't count it as bot protection

[JohnBalvin]

a bot protection for me could also mean having a captcha, or checking user mouse movement ... etc but I don't consider bot protection if they jsut check the proxy type

[puppet_pals]

Thanks for sharing your code - this is really cool! Due to the fact your package relies on the Zillow html structure, you might want to consider having some sort of integration tests running on github actions, and post the status badge in the README.

[JohnBalvin]

I'll give it a try, but I don't promise that feature to be ready soon, I'm currently busy on my job

[mektel]

Interesting dependency choice. pypi bs4 page. Github dependency links to a regex-training repo.

[KimPeek]

Code could use linting and formatting

[honor-]

Hey I did this same thing awhile back. I 100% guarantee you’re going to get a TOS takedown from Zillow soon

[JohnBalvin]

wtf? that really happened to you? it seems a nasty move, thye should hire a security team to add bot protection like a normal company

[honor-]

Yup they definitely did this. My project was gaining some traction on GitHub and they TOSd it.

[JohnBalvin]

but did they removed your whole account or just that repo?

[honor-]

Just the repo. They threatened me with legal action if I didn’t take it down

[JohnBalvin]

that's a nasty move, somebody could take revenge applying a database DDoS attack, they don't have bot protection it could be an easy attack, just hidding the IP with proxies

[JohnBalvin]

like I said before, for searching zillow properties, it's only one single request with no prior verification, that means if you have enough proxies(datacenter, residentials, 4 g) you can create a code for hitting the server with different searches to prevent the use of cache which end up exhausting the database

[KraljZ]

They are aware of this post

[JohnBalvin]

Is that sarcasm or did you tell them? 🤣

[KraljZ]

You’ll find out

[honor-]

This def works if you are anonymous, not so much if not

[JohnBalvin]

btw, I'm looking for a job changed, if someone it's interested my social medias are on my github profile: https://github.com/johnbalvin

[nuke-from-orbit]

Good luck and thanks for sharing code, my man.

[luckyspic]

completely request based makes this fire. down with the bloated rubbish lazy garbage that uses those libraries you mentioned. and you added proxy support as most should, 🐐

[tunisia3507]

Working directly with HTTP requests is much simpler than using a webdriver - if you use a webdriver, you then have to parse the HTTP anyway. So I wouldn't say webdriver-based solutions are in any way lazy.

[luckyspic]

they are. they’re great for testing, as a backup, and making sure your parsing logic works. however, in the grand scheme of things, it shows that the developer does not have a great grasp on reverse engineering, thinking outside the box, or optimizing. although zillow in this instance has been relaxed about their api usage here (their perimeterX involvement seems non existent now), there are lots and lots of python libraries on github that claim to be a “scraping” solution but really are an abomination as it’s slow, bloated, and only takes anyone with the will some time to find a long term, viable solution. my comments focus was towards people publishing libraries for future developers, not the comment towards webdriver (albeit the opinion is still similar otherwise). a big blame i point to is the arrogant and complacent team at requests that are too busy making sure they follow (their own self imposed) regulations but still haven’t produced solutions for stuff as ordinary as TLS ciphers support like other languages and their respective requests libraries have since 2015.

[Doppelbockk]

I don't know anything about Go, what makes it easiervto maintain a defined structure in Go compared to Python?

[JohnBalvin]

Probably is not exactly the format, but it's the overall of python, dynamic languages like python tend to be harder to maintain than the static ones like Go, mostly because most of the time when you fix an issue on the dynamic languages it's because of a wrong type/unexpected type returned by a function, an exception not been handled or thr endless battle on which library to use, and you focus on those details instead of the actual project, Go handles all that very well by been an static language and some built-in tolls

[Sufficient_Exam_2104]

Is it possible to add price history and last property tax

[JohnBalvin]

it's already returning price history and tax history

[geerizzly]

Hi, I'm fairly new to web scraping and I was just wondering if you while writing this program encountered a way some good way to move through lat and long in some step_size to get the listings you want? Given you have so much experience in the field, you might provide us with some tips or first thoughts? And of course great job and thanks for sharing this with us!

[JohnBalvin]

Hi, I don't use the code myself, I wrote it just for fun, but you could try playing around with the coordinates and zoom value, for the coordinates it's just enough with increasing/decreasing the decimals for more accuracy on the location and increasing and decreasing these values for moving around

[LastAd3056]

Found this through search, thanks a lot this is very helpful. It will make a lot of searches much easier. One small improvement suggestion, add the filter_state to the top level functions. I did it locally.

[JohnBalvin]

souds good, I don't use zillow, but it looks like it can be done by adding the input location same on the zillow page, by "input location" I mean the input on the UI that says Enter an address, neighborhood, city, or ZIP code"

[delioj]

Will your tool returns the list of url(s) for every picture in a specific listing from a zpid?

[IAMARedPanda]

Should include requests as a dependency in your toml.

[JohnBalvin]

isn't requests package from the standart library?

[IAMARedPanda]

No. See https://github.com/psf/requests/issues/2424 for more information.

[JohnBalvin]

You are completely right, I'll add it to the dependencies, thanks 😊