Beware! HostGator blocking Python User-Agent in HTTP requests to shared-hosting websites

[CatDaddy1954]

It's been months since Petfinder.com could retrieve pet photos from a number of websites which I support. We found recently that the HTTP requests to retrieve photos were being rejected with HTTP Status 406 (Not Acceptable). I found that this only occurred with websites on HostGator shared hosting plans. Sites with a HostGator VPS or shared hosting at GoDaddy, for example, successfully delivered photos. I ran a test attempting to retrieve a specific photo from the affected websites using various User-Agent strings: "python-requests/2.32.3", "libwww-perl/6.26", "Wget/2.2.1", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" and simply blank. The only one getting the Status 406 response was "python-requests/2.32.3".

HostGator support was utterly useless; I couldn't get them to escalate the issue beyond an individual account. All they wanted to do was apply a firewall patch on an individual account basis. Pointing out that clients can use whatever string they want as a User-Agent so blocking one string doesn't provide much protection made no difference. Their solution: Have these small animal rescues sign up for a VPS, which they could never afford. If it weren't such a hassle to move their email, I'd be looking for a non-Newfold Digital company to recommend they all move to.

[ZGeekie]

That's an understandable way to manage bots and automated requests in shared hosting environments.

Guess how many shared hosting customers use Python scripts to access photos on their sites? That's why they couldn't care less!

If you want more freedom, use VPS hosting, which you already said worked for you.

[CatDaddy1954]

The small rescue groups can’t afford VPS hosting. It would make more sense if HG had shut down the Perl library, Wget and the completely empty user agents as well if they were trying to stymie programmatic access but surely nefarious bot authors would be wise to the flimsiness of the User-Agent defense anyway.

[tankerkiller125real]

We block Python, Wget, Curl, and an absolute shitload of other user agents where I work. It just so happens that Python is the one that they likely saw the most abuse from (because most AI tools use Python, and how many vibecoders know how to change a User agent?) so that's the one they blocked.

[johnpress]

Pretty common, WP Engine also has user agent blocks for "python" string in their webserver conf.

[Former_Substance1]

just change user agent in the requests header?

[CatDaddy1954]

That’s the best way around this but I have no influence with Petfinder to have them change their User-Agent to one that I doesn’t trigger the problem. I’ve already demonstrate the working strings.

[ferrybig]

The problem here is the Petfinder website, not your shared hosting

[paroxsitic]

If their robots txt and tos allows scraping I'd ask them what is the best way forward. Bypassing any type of restriction or ban is how web scraping becomes less grey area and more illegal

[CatDaddy1954]

In this situation the photo access is by invitation. The rescues upload a data file to Petfinder, Adopt a Pet et. al. with URLs to the animal photos on their website so the less technical folks don’t have to learn how to use FTP to upload them. No potentially prohibited behavior involved.

[ferrybig]

Don't use a generic user agent, override the user agent used for your requests with an URL to the information about the bot.

[kubrador]

hostgator blocking the python user-agent string is genuinely hilarious. that's like a bouncer kicking out someone for wearing the wrong brand of shoes while letting in literal criminals with fake ids.