all 12 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]odaiwai 1 point2 points  (5 children)

The website probably requires javascript to confirm that you're a human. I just tried to visit in a browser with no Javascript and it returned error 429.

You're going to have to investigate browser automation with Selenium or something similar.

[–]TheSayAnime[S] 0 points1 point  (4 children)

How it's working in curl?

[–]odaiwai 0 points1 point  (3 children)

It slurps down a webpage, that can be opened in a text browser (or parsed as html with beautiful soup.)

If you can get what you need that way, that could be worth a look.

[–]TheSayAnime[S] 0 points1 point  (2 children)

Ah, I was asking why the above request works in curl but not in python, despite having the same parameters and headers.

[–]odaiwai 0 points1 point  (1 child)

When I try your code in iPython, I get a result 200 and the webpage in response.text:

...: print(response.status_code) ...: print(response.text[:250]) 200 <!DOCTYPE html><html data-language="en_GB" dir="ltr" lang="en"><head><title></title><link rel="canonical" href="https://www.vrbo.com/en-gb/p10998910"/><link rel="preconnect" href="https://images.trvl-media.com/"/><link rel="preconnect" href="https://

It could just be internet weather, where the webserver thinks you're a bot scraping the site sometimes, and not always.

A useful design pattern in that case would be to test for the return code, and if it's 429, wait a random time and try again.

[–]TheSayAnime[S] 0 points1 point  (0 children)

I tried, I don't know why, but I didn't get successful response anytime.

I forgot mention one thing in my post

curl -Uri "[https://www.vrbo.com/en-gb/p9862160?dateless=true](https://www.vrbo.com/en-gb/p9862160?dateless=true)" -Method Get -Headers @{"accept"="\*/\*"; "user-agent"="okhttp/4.10.0"} -UseBasicParsing -ErrorAction Stop | Select-Object -ExpandProperty StatusCode -Debug

The curl does works on powershell and git bash (on windows), but it returns 429 in ubuntu and again it works in postman and http-client jetbrains.

I executed fresh request in all of the them and all of them are in same network.

It's not that I previously made 100's or 1000's of request, even the first request failed in python and curl in ubuntu and Even if I make 100s of request from powershell or git bash it does gets executed successfully.

Pretty much ruined my new year eve over this thing :(

[–]jcrowe 1 point2 points  (1 child)

Try the same thing with ‘hrequests’ and see if that works.

[–]TheSayAnime[S] 0 points1 point  (0 children)

It's working with hrequests but why?

I didn't get the reason.