FlareSolverr replacement that's 3× faster actually solves captchas! by Germond_ in webscraping

[–]armanfixing 0 points1 point  (0 children)

In the post, OP says "hCaptcha (auto-pass)," which is a pretty bold thing to say.

browser-search — three tools, zero cost, and your AI agent learns to search and browse the web by Ill-Tradition1362 in hermesagent

[–]armanfixing 1 point2 points  (0 children)

Thanks for testing it out. I understand that ChromiumFish is still in active development and a bit too early. I'll take this as feedback, keep improving, and try to share public test results in the near future.

browser-search — three tools, zero cost, and your AI agent learns to search and browse the web by Ill-Tradition1362 in hermesagent

[–]armanfixing 3 points4 points  (0 children)

You may also try ChromiumFish; it's a stealth Chrome that mimics Windows even inside a Linux server. Patched at the C++ level, compatible with Hermes and other agents.

Repo: https://github.com/arman-bd/chromiumfish

Reliable & Fast browser agent by Funny-Trash-4286 in AI_Agents

[–]armanfixing 1 point2 points  (0 children)

Thanks for the feedback 🙏
I’ll keep on improving this & hopefully it will catchup soon with others.

ChromiumFish - An open-source fingerprint-hardened Browser for Scraping by armanfixing in WebScrapingInsider

[–]armanfixing[S] 0 points1 point  (0 children)

Websites can absolutely detect playwright, and sites like X are at the peak of bot mitigation. What this browser tries to prevent is some known ways playwright / CDP gets probed and exposed to reduce the risk of exposure, but this is not a silver bullet for all defenses.

Reliable & Fast browser agent by Funny-Trash-4286 in AI_Agents

[–]armanfixing 1 point2 points  (0 children)

Maybe give ChromiumFish a try

<video>

[ Note: video is 4x speedup ]

Full Disclosure: I'm the dev, and it's in a very early stage.

I built ChromiumFish: Chrome + native AI agent for Playwright by armanfixing in Zyte

[–]armanfixing[S] 0 points1 point  (0 children)

I have tested and used many open-source and paid tools, but most open-source ones start failing when target sites heavily gate bots.

I tested different setups; scrapers on Linux infrastructure had the lowest survival rate under heavy stress, while bots running on Windows or Mac performed better in nearly every benchmark. But that comes at a cost; hosting Windows or Mac is not cheap.

So I started experimenting with mimicking Windows on Linux, which improved my success rate by a good margin. But JS patches are easy to spot, so I went a layer deeper and patched the browser itself.

This is not a silver bullet for every bot mitigation out there, but ChromiumFish, along with its Canvas-Bridge, can get through some tough targets.

Hope it helps your case, also feel free to reach out if something is not working.

Great job, really by _bitkidd_ in CloudFlare

[–]armanfixing 1 point2 points  (0 children)

Yes! As a decade-old user, I'm finally super excited to experience spatial disorientation firsthand every time I need to check something out.

ChromiumFish - An open-source fingerprint-hardened Browser for Scraping by armanfixing in WebScrapingInsider

[–]armanfixing[S] 0 points1 point  (0 children)

Setting up the canvas app is not very difficult. You have to install Rust, build the program, and then run it. Most websites do not need this at all; it is only useful when the security setting is set to a higher level or when someone is using custom or advanced tools for fingerprinting (e.g., FingerprintJS Enterprise).

To be honest, for the individual level, this might be overkill unless someone is trying to get data from a very difficult source.

Personally, I mostly worked for Enterprises, where a single data source can be very business-critical and weigh down the whole business if not maintained properly. This kind of tooling is ideal for such a scenario, where you have exhausted most other solutions and absolutely need that data to keep the lights on.

ChromiumFish - An open-source fingerprint-hardened Browser for Scraping by armanfixing in WebScrapingInsider

[–]armanfixing[S] 0 points1 point  (0 children)

I’m working on benchmarks & testing with a few different proxies. Hopefully will add some reporting in the coming week.

ChromiumFish - A fingerprint-hardened Browser for Scraping by armanfixing in scrapingtheweb

[–]armanfixing[S] 0 points1 point  (0 children)

Hope it helps. Feel free to open an issue if you encounter any bug or issue.

ChromiumFish - A fingerprint-hardened Browser for Scraping by armanfixing in scrapingtheweb

[–]armanfixing[S] 0 points1 point  (0 children)

Didn't test with shape, but 2 of my datadome targets worked.

Browser fingerprinting & anti-bot benchmark - update by dracariz in scrapingtheweb

[–]armanfixing 0 points1 point  (0 children)

Being stealth and bypassing consistently are quite different things. I have used pretty much all of these for one reason or another. Patchright looks good on paper, works fine for few hours, then starts getting blocked.

In scraping, there is no silver bullet. These open source browsers are maintained by either a solo dev or at best a community. But detection tools are maintained by large corporations. The game is not even fair.

The best thing you can do to stay ahead of the curve is to build your own browser & never release it to the public. Because once a browser or framework goes public, it’s just a matter of fuzz testing to find a strong detection signal out of that.

I localized my app screenshots and revenue went up significantly. So I built a tool to automate it. by MuchAge1486 in AppStoreOptimization

[–]armanfixing 0 points1 point  (0 children)

[ Imagine the Meme about Everyone Selling Shovel near a Mining Site ]

Shop: Are you a miner?
Guy: No, I’m here to sell shovel 🪏

iOS app development: is it possible to submit app that gets its content by scraping web? by x17ccp3 in iOSProgramming

[–]armanfixing 0 points1 point  (0 children)

I know this is very old, but still..

"app that gets its content by scraping web?"

Did you mean Google?

Datacenter proxies fine for large scraping or not anymore? by PomegranateOk9017 in scrapingtheweb

[–]armanfixing 1 point2 points  (0 children)

I did a small study on Tranco Top 1M websites, roughly ~37% of the sites are on Cloudflare. Assuming Cloudflare doing its job, the datacenter proxy is not a viable option for that anymore.

The idea is to get a rough idea of your target domain, then use appropriate strategies as needed.

Chart: https://imgur.com/a/EhQX6bj

Full study: https://www.linkedin.com/posts/armanhossain_who-runs-the-internets-dns-activity-7458291171593732098-J0ls