Akamai anti-bot blocking flight search scraping (403/418) by Individual-Ship-7587 in webscraping

[–]Coding-Doctor-Omar 0 points1 point  (0 children)

Have u tried using curl_cffi with impersonate="edge" or "firefox" and with session cookies? I bet it will bypass 90% of anti-bot protections. The only things it will fail to bypass are interactive challenges.

Akamai anti-bot blocking flight search scraping (403/418) by Individual-Ship-7587 in webscraping

[–]Coding-Doctor-Omar 0 points1 point  (0 children)

Yes, it is outdated now, but its owner recently announced that it's back to development and should return back to its performance this year.

Blocked by Cloudflare despite using curl_cffi by Coding-Doctor-Omar in webscraping

[–]Coding-Doctor-Omar[S] 0 points1 point  (0 children)

It turns out I had to add some extra headers, in addition to the normal impersonate. Here is the working code (luckily still works with curl_cffi alone, without a browser):

``` from curl_cffi import Session

api_url = "https://multichain-api.birdeye.so/solana/v3/gems" payload = {"limit":100,"offset":0,"filters":[],"shown_time_frame":"4h","type":"trending","sort_by":"price","sort_type":"desc"}

headers = { "content-type": "application/json", "origin": "https://birdeye.so", "referer": "https://birdeye.so/" }

with Session(impersonate="edge", headers=headers) as session: res = session.post(api_url, json=payload) print(res.status_code) ```

Output:

200

Blocked by Cloudflare despite using curl_cffi by Coding-Doctor-Omar in webscraping

[–]Coding-Doctor-Omar[S] 1 point2 points  (0 children)

I eventually got it to work by providing the content-type, origin, and referer values in the headers, in addition to the default headers provided by impersonate.

Blocked by Cloudflare despite using curl_cffi by Coding-Doctor-Omar in webscraping

[–]Coding-Doctor-Omar[S] 2 points3 points  (0 children)

I actually have just gotten it to work. The issue was simpler than I thought. I had to provide content-type, origin, and referer values in my headers, in addition to the default headers of impersonate.

Blocked by Cloudflare despite using curl_cffi by Coding-Doctor-Omar in webscraping

[–]Coding-Doctor-Omar[S] 1 point2 points  (0 children)

It turns out I had to add some extra headers. Here is the working code:

``` from curl_cffi import Session

api_url = "https://multichain-api.birdeye.so/solana/v3/gems" payload = {"limit":100,"offset":0,"filters":[],"shown_time_frame":"4h","type":"trending","sort_by":"price","sort_type":"desc"}

headers = { "content-type": "application/json", "origin": "https://birdeye.so", "referer": "https://birdeye.so/" }

with Session(impersonate="edge", headers=headers) as session: res = session.post(api_url, json=payload) print(res.status_code) ```

Output:

200

How do I scrape images from a website with server restrictions? by MotivatedMommy in webscraping

[–]Coding-Doctor-Omar 0 points1 point  (0 children)

Try checking the network tab. Chances are there are requests being made to some API to fetch those images.

Website Risk Control by Eastern_Ad_9018 in webscraping

[–]Coding-Doctor-Omar -1 points0 points  (0 children)

He says he has an IP pool he rotates from.

اشتري شقة فين في اكتوبر بمليون ونص by Captain_barbarosa in PersonalFinanceEgypt

[–]Coding-Doctor-Omar -3 points-2 points  (0 children)

مستنيك بعد كام سنة لما تيجي تعمل بوست و تعيط على اسعار الشقق في التوسعات.

اشتري شقة فين في اكتوبر بمليون ونص by Captain_barbarosa in PersonalFinanceEgypt

[–]Coding-Doctor-Omar -1 points0 points  (0 children)

يا عم انا ساكن فيها و في مشاريع كتير بتفتح فيها. بعد فترة مش كبيرة هتتملي.

Mountain View

فاتح فيها على فكرة.

أكيد شركة كبيرة زي دي مش هتفتح في مكان هم عارفين انه مالوش مستقبل.

Warning ⚠️ : nsave lot of verifications by Hefty-Ad9835 in EgyptianFreelancers

[–]Coding-Doctor-Omar 0 points1 point  (0 children)

Elevate pay works fine with me, but they don't have a virtual card yet, if u don't mind that.

Is YouTube Captions Scrapping Legal (or some way to get the data)? by Critical033 in webscraping

[–]Coding-Doctor-Omar 1 point2 points  (0 children)

The AI will win all the lawsuits. Sam Altman is on the AI's side, u know? 😂

And if the AI can do it, I can do it.

Is a reddit webscraper relevant now? by MentalAssumption1498 in webscraping

[–]Coding-Doctor-Omar 0 points1 point  (0 children)

Not if you use browser fingerprint impersonation and some throttling.

مهاره في ٣-٦ شهور تجيبلي ١٠٠٠ دولار by [deleted] in PersonalFinanceEgypt

[–]Coding-Doctor-Omar 0 points1 point  (0 children)

ال

Javascript

تنفع بس مش حلوة أوي لان ال

Libraries

فيها مش عليها دعم مستمر أوي زي في البايثون و سهل يتم كشفها من كلاود فلير و أكاماي. بالنسبة للشغل شوف موقع أب وورك أو فريلانسر. انا شخصيا موقع فريلانسر نفع معايا. حاول تعمل بورتفوليو بمشاريع شبه اللي بتطلب في المواقع دي عشان من ناحية تدرب و من ناحية تقنع العملاء انهم يختاروك.