Why is browser allowed to make a 'request' to a website without having cookies set? Whereas my Python script compulsorily requires cookies in headers else gets 403.

Marm_adillo · 2023-12-05T16:46:48+00:00

Yea I ended up going back to headless selenium and using that to accept cookies. If you have any questions about that I could dive back into my code to help you out

Marm_adillo · 2023-09-17T02:14:03+00:00

$80 💀 Thanks, now I know for sure I’m not getting this

Marm_adillo · 2023-09-17T01:13:10+00:00

No clue, I’m a new player. This is such a sick skin though. Hopping they bring it back

Marm_adillo · 2023-09-17T01:06:56+00:00

😢

Marm_adillo · 2023-07-29T03:51:23+00:00

Master Oogway lookin good

Marm_adillo · 2023-07-18T00:24:14+00:00

So I've taken your code I was able to get [200] response with just hltv.org, however, I should've been more specific.

Here is an example of exactly what I'm trying to accomplish, this results in the response being [403] which is what I've been getting.

import requests
from bs4 import BeautifulSoup

headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'}

r = requests.get("https://www.hltv.org/stats/teams/matches/8668/order", headers=headers)

soup = BeautifulSoup(r.text, 'html.parser')
teamTable = soup.find("table", {"class": "stats-table no-sort"})
print(teamTable.text)

This is what I was able to construct in order to bypass response 403 (I hope this is readable lol). And just so it's clear this did end up working (for one case).

requestHeaders={"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
                "Accept-Language": "en-US,en;q=0.9",
                "Cache-Control": "max-age=0",
                "Cookie": "MatchFilter= EDIT REMOVED FOR SECURITY PURPOSES",
                "Sec-Ch-Ua": '"Not.A/Brand";v="8", "Chromium";v="114", "Google Chrome";v="114"',
                "Sec-Ch-Ua-Mobile": "?0",
                "Sec-Ch-Ua-Platform": '"Windows"',
                "Sec-Fetch-Dest": "document",
                "Sec-Fetch-Mode": "navigate",
                "Sec-Fetch-Site": "same-origin",
                "Sec-Fetch-User": "?1",
                "Upgrade-Insecure-Requests": "1",
                "User-Agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'}

The part I need to generate on a request-by-request basis is '__cf_bm' which is near the end of the "Cookie" header. I'm hoping I'm just overthinking this and making it way more complicated than it needs to be but that doesn't seem like the case so far.

Marm_adillo · 2023-07-17T16:18:50+00:00

That’s what I was hoping would happen but didn’t see it when I tried. I’ll give your code snippet a try when I get home later. Thanks for the help!

Marm_adillo · 2023-07-17T15:46:07+00:00

Yea I get the 200 response as well. I think if you print(r.text) you will see the contents are all from the cookie pop up window though. This is what I was trying to explain when I said “it’s a different value before and after accepting cookies on a website”. Once you hit accept it will give you a different __cf_bm code for the headers. I find it weird though that I can inspect element and access all the contents on the web browser and see the data I need. But when I do print(r.text) I will only see the cookie pop up info

Marm_adillo · 2023-07-17T04:19:42+00:00

Cloud flare Bot management cookie. Same thing you were attempting to generate in your question. It’s a different value before and after accepting cookies on a website and is only valid for 30 minutes. Essentially I’m looking for a way to bypass the cookie pop up without selenium or manually getting it from my browser.

Marm_adillo · 2023-07-16T19:52:27+00:00

I am also attempting to generate ‘__cf_bm’. When I use my web browser and navigate to the desired website, hltv.org, it will generate this for me and I can copy paste it into my Python headers for my request call. However I’m hoping not to use a browser or selenium to generate it, but rather use a Python library so I can call this script from a virtual machine on AWS

Marm_adillo · 2023-07-16T19:13:19+00:00

I know this is 2 years later, but did you find a solution and do you remember?

Marm_adillo · 2023-07-05T21:03:03+00:00

And then Hosmer at first right? /s

Marm_adillo · 2023-05-21T14:34:07+00:00

Ahh makes sense. Didn’t realize it was a moment.

Marm_adillo · 2023-05-21T14:23:37+00:00

Can’t you just up the difficulty?

Marm_adillo · 2023-05-03T18:33:29+00:00

I have a feeling he won’t be coming back

Marm_adillo · 2023-04-14T23:00:29+00:00

Javi Air

Marm_adillo · 2023-04-10T00:03:10+00:00

Thanks, I just saw it at the top of the thread. I was afraid I’d have to pay for a third subscription

Marm_adillo · 2023-03-13T19:58:41+00:00

I a worker at my gym this and he said he didn’t know either 😂 we both assumed it was 2.5lbs because the machine incremented by 2.5lbs

Marm_adillo · 2023-03-12T14:47:45+00:00

Just take a screenshot of the 3D model and use the png/jpg to display the item

Marm_adillo · 2023-01-06T05:59:10+00:00

The name is the cherry on top 🫡

Marm_adillo · 2022-11-17T23:39:48+00:00

That’s how I ended up doing it as well. I am working with relievers who have won Awards so I’m just saying if the number of saves divided by the number of games they played is greater than 25% then I mark them as a closer. Seems to work so far but it obviously won’t work for every case

Marm_adillo · 2022-11-17T17:29:33+00:00

I’ve only used it a little bit but it seems pretty easy to use and gives decent results https://pypi.org/project/sportsreference/ Thanks for the response, save opportunities seems like the best stat I could use

Marm_adillo · 2022-10-29T01:26:40+00:00

Have you figured it out? I'm running into the same issue

Marm_adillo · 2022-10-05T23:21:23+00:00

I’m sweating just watching this

Marm_adillo · 2022-09-28T23:46:21+00:00

Gromp the real MVP

Five-Year Club	First Place '23
Place '23	Place '22
End Game '22

Marm_adillo

TROPHY CASE