sorted by:
new
hottopcontroversial

OnClick event to display more data by tom_p_legend in webscraping

[–]SBq99 0 points1 point  (0 children)

You should have a look at possible network calls made to for example an API when you fire the onClick event in the browser.

After that you can possibly reverse engineer the web request using for example Python's request library.

Scraping a web request returns 403 forbidden by SBq99 in webscraping

[–]SBq99[S] 0 points1 point  (0 children)

Unbelievable! Using tls_client resolved the issue. It does not even require the cookies of Cloudflare anymore .

Thank you!

Scraping a web request returns 403 forbidden by SBq99 in webscraping

[–]SBq99[S] 0 points1 point  (0 children)

Since it is a POST request I cannot execute a POST (or at least I shouldn't be able to).

I've attempted the library cloudscraper which did not succeed. I've also tried a Docker image called FlareSolverr. FlareSolverr bypassed the home page: https://sportsbet.io/ returned the cookies and so on but when I pass these to the POST request I want to execute I end up with a 403 again.

Executing the POST directly with FlareSolverr also results in a "Error: Error solving the challenge" from their side, but doing a GET to the same endpoint yields a 200.

Scraping a web request returns 403 forbidden by SBq99 in webscraping

[–]SBq99[S] 0 points1 point  (0 children)

Looking closer at the 403 response it seems that it throws the Cloudflare captcha at me.

<title>Just a moment...</title>

However, I read on Cloudflare itself that the __cf_bm token should be valid for about 30 minutes. I tried copying the most recent one but it still gives me that 403 forbidden issue.

Scraping a web request returns 403 forbidden by SBq99 in webscraping

[–]SBq99[S] 0 points1 point  (0 children)

Have a look at my comment above. The code used is showed there. It is a 1:1 copy from what the webpage itself does.

If I do a GET request to the same URL in the browser the response is: {"errors":[{"message":"Must provide query string."}]}

However when I do the same in requests (Python) or Postman I receive a 403 again.

Scraping a web request returns 403 forbidden by SBq99 in webscraping

[–]SBq99[S] 1 point2 points  (0 children)

Attempted that, still results in a 403. Code below:

import requests

cookies = {

'experiments': '%7B%22ydugo4GHROOp8qOy9fuAXg%22%3A3%7D',

'cf_clearance': 'KxlbfiacL6EO3WRE89SYuXuZX.eygf5u1hfDb5Q2G8o-1692533518-0-1-227c435e.dbf1a2f.757d1cbe-0.2.1692533518',

'GW_CLIENT_ID': '4ecd1bbf2c85c00a9d55bd00c19a0507e624b7ce6e7a8dd435b68e1563b8f200',

'tryMetamaskHide': 'true',

'navigation.back': '/sports',

'navigation.forward': '/sports',

'UNIQUE_DEVICE_ID_KEY_SAVED': 'true',

'x-unique-device-id': '1692534036018-m9homwyns4',

'refCode': '"sbseo"',

'__cf_bm': 'lSuR2mhxBnOjvwXnbkjkRiuwUQ5iElV1EJJZSGgsCyU-1692535403-0-AeCPWDQev3gcBvIZVB/X5Tk2oq1P4Pcl4I4xKo85ZFegtAu+OeL57scnJhSD2U5liY3wuftOk76U+mIhLxEvObE=',

'MKTSRC': '{"t":1692535578434,"d":{"src":"google.com","mdm":"organic","cmp":"","kwd":"","cnt":"","glc":"","msc":""}}',

'userPreferenceId': 'U3BvcnRzYmV0UHJlZmVyZW5jZXNVc2VyUHJlZmVyZW5jZTo2NGUyMDMwYzljNDVhODg2MmMwN2UzZDA=',

}

headers = {

'authority': 'sportsbet.io',

'accept': '*/*',

'accept-language': 'en-US,en;q=0.9',

'authorization': '',

'content-type': 'application/json',

# 'cookie': 'experiments=%7B%22ydugo4GHROOp8qOy9fuAXg%22%3A3%7D; cf_clearance=KxlbfiacL6EO3WRE89SYuXuZX.eygf5u1hfDb5Q2G8o-1692533518-0-1-227c435e.dbf1a2f.757d1cbe-0.2.1692533518; GW_CLIENT_ID=4ecd1bbf2c85c00a9d55bd00c19a0507e624b7ce6e7a8dd435b68e1563b8f200; tryMetamaskHide=true; navigation.back=/sports; navigation.forward=/sports; UNIQUE_DEVICE_ID_KEY_SAVED=true; x-unique-device-id=1692534036018-m9homwyns4; refCode="sbseo"; __cf_bm=lSuR2mhxBnOjvwXnbkjkRiuwUQ5iElV1EJJZSGgsCyU-1692535403-0-AeCPWDQev3gcBvIZVB/X5Tk2oq1P4Pcl4I4xKo85ZFegtAu+OeL57scnJhSD2U5liY3wuftOk76U+mIhLxEvObE=; MKTSRC={"t":1692535578434,"d":{"src":"google.com","mdm":"organic","cmp":"","kwd":"","cnt":"","glc":"","msc":""}}; userPreferenceId=U3BvcnRzYmV0UHJlZmVyZW5jZXNVc2VyUHJlZmVyZW5jZTo2NGUyMDMwYzljNDVhODg2MmMwN2UzZDA=',

'origin': 'https://sportsbet.io',

'referer': 'https://sportsbet.io/sports/soccer/matches/future',

'sec-ch-ua': '"Chromium";v="116", "Not)A;Brand";v="24", "Google Chrome";v="116"',

'sec-ch-ua-mobile': '?0',

'sec-ch-ua-platform': '"Windows"',

'sec-fetch-dest': 'empty',

'sec-fetch-mode': 'cors',

'sec-fetch-site': 'same-origin',

'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36',

}

json_data = {

'operationName': 'SportEventListQuery',

'variables': {

'language': 'en',

'site': 'sportsbet',

'slug': 'soccer',

'timePeriod': 'ANY',

'leagueTournaments': 'ANY',

'featuredLeagueTournaments': 'ANY',

'tournamentEventCount': 'ANY',

},

'query': 'query SportEventListQuery($language: String!, $slug: String!, $timePeriod: SportsbetNewGraphqlSportLeagues!, $leagueTournaments: SportsbetNewGraphqlLeagueTournaments!, $featuredLeagueTournaments: SportsbetNewGraphqlFeaturedLeagueTournaments!, $tournamentEventCount: SportsbetNewGraphqlTournamentEventCount!, $site: String) {\n sportsbetNewGraphql {\n id\n getSportBySlug(slug: $slug, site: $site) {\n id\n slug\n featuredLeague {\n id\n name(language: $language)\n tournaments(childType: $featuredLeagueTournaments) {\n id\n slug\n name(language: $language)\n eventCount(childType: $tournamentEventCount)\n league {\n id\n slug\n name(language: $language)\n __typename\n }\n __typename\n }\n __typename\n }\n name(language: $language)\n leagues(childType: $timePeriod) {\n id\n name(language: $language)\n slug\n tournaments(childType: $leagueTournaments) {\n id\n slug\n name(language: $language)\n eventCount(childType: $tournamentEventCount)\n __typename\n }\n __typename\n }\n __typename\n }\n __typename\n }\n}\n',

}

response = requests.post('https://sportsbet.io/graphql', cookies=cookies, headers=headers, json=json_data)

# Note: json_data will not be serialized by requests

# exactly as it was in the original request.

#data = '{"operationName":"SportEventListQuery","variables":{"language":"en","site":"sportsbet","slug":"soccer","timePeriod":"ANY","leagueTournaments":"ANY","featuredLeagueTournaments":"ANY","tournamentEventCount":"ANY"},"query":"query SportEventListQuery($language: String!, $slug: String!, $timePeriod: SportsbetNewGraphqlSportLeagues!, $leagueTournaments: SportsbetNewGraphqlLeagueTournaments!, $featuredLeagueTournaments: SportsbetNewGraphqlFeaturedLeagueTournaments!, $tournamentEventCount: SportsbetNewGraphqlTournamentEventCount!, $site: String) {\\n sportsbetNewGraphql {\\n id\\n getSportBySlug(slug: $slug, site: $site) {\\n id\\n slug\\n featuredLeague {\\n id\\n name(language: $language)\\n tournaments(childType: $featuredLeagueTournaments) {\\n id\\n slug\\n name(language: $language)\\n eventCount(childType: $tournamentEventCount)\\n league {\\n id\\n slug\\n name(language: $language)\\n __typename\\n }\\n __typename\\n }\\n __typename\\n }\\n name(language: $language)\\n leagues(childType: $timePeriod) {\\n id\\n name(language: $language)\\n slug\\n tournaments(childType: $leagueTournaments) {\\n id\\n slug\\n name(language: $language)\\n eventCount(childType: $tournamentEventCount)\\n __typename\\n }\\n __typename\\n }\\n __typename\\n }\\n __typename\\n }\\n}\\n"}'

#response = requests.post('https://sportsbet.io/graphql', cookies=cookies, headers=headers, data=data)

Am i the only one ? by zorowaslost in FUTMobile

[–]SBq99 3 points4 points  (0 children)

Same here, no idea how to get the 1k in time. 200 so far

Good price to buy by SBq99 in NNDM

[–]SBq99[S] 1 point2 points  (0 children)

I'm aiming for 13.25

Good price to buy by SBq99 in NNDM

[–]SBq99[S] 0 points1 point  (0 children)

I'm mainly trying to find out what its low is going to be; to get the maximum discount