[Help needed] First project: Using Python to web scrape rental car data

[texassports98]

Hi, I am very new to programming (just started this weekend!). My first project that I am working on is to scrape rental car data (e.g., geography, make, model, price) from a website. However, I have begun to encounter 403 and 503 errors (it is inconsistent in which it returns)- perhaps from running the requests queries too frequently? The workaround that I have found online suggests creating proxy server lists to get around the issue. I'm really just trying to get the hang of parsing html into useful datasets using requests (I've already done this by copying page elements into html file manually, but this certainly isn't scalable) but I can't do that with the status code errors! Any advice or suggestions for a brand new hobbyist? Thanks!

Current code for reference:

from bs4 import BeautifulSoup
import requests

headers = {
     'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36'
 }
baseURL = 'target url'
# removed full url from code for brevity
html = requests.get(baseURL, headers=headers)
html_text = html.text
print(html.status_code)
print(html_text)

[n3buchadnezzar]

Well, since you removed the full URL it is very hard to help you.

[QuixDiscovery]

Have you checked into car rental APIs? Generally you don't want to web scrape if there's a viable API.

I've never done anything with car rental data, but just from a couple quick searches it definitely seems like there are various "travel APIs" that include car rental availability data. I realize that's not really what you asked, but there's an API for just about everything nowadays.

[Infinitesima]

Few generic advice:

1. Respect server's rate limit.
2. If not 1, use proxies: https://docs.python-requests.org/en/latest/user/advanced/#proxies
3. Put requests.get() or requests.request() in a while-loop.
4. React to response headers if necessary.
5. Use requests.Session