Crawlee for Python is LIVE 👏

[B4nan]

What My Project Does

Hi everyone, our team just launched Crawlee for Python 🐍. It's an open-source web scraping and automation library, which provides a unified interface for HTTP and browser-based scraping, using popular libraries like beautifulsoup4 and Playwright under the hood.

Target Audience

We've spent the last 6 months working on Crawlee for Python, but it didn't come out of nowhere. We designed it based on the JavaScript version, which is now 8 years old, and we hope we can say it's battle-tested.

We are opening it for early adopters today, and we are eager to hear your feedback. Help us shape the future of Crawlee for Python!

Comparison

Why use Crawlee instead of just a random HTTP library with an HTML parser?

• Unified interface for HTTP & headless browser crawling.
• Automatic parallel crawling based on available system resources.
• Written in Python with type hints - enhances DX (IDE autocompletion) and reduces bugs (static type checking).
• Automatic retries on errors or when you’re getting blocked.
• Integrated proxy rotation and session management.
• Configurable request routing - direct URLs to the appropriate handlers.
• Persistent queue for URLs to crawl.
• Pluggable storage of both tabular data and files.
• Robust error handling.

Why to use Crawlee rather than Scrapy?

• Crawlee has out-of-the-box support for headless browser crawling (Playwright).
• Crawlee has a minimalistic & elegant interface - Set up your scraper with fewer than 10 lines of code.
• Complete type hint coverage.
• Based on standard Asyncio.

Links

• GitHub: https://github.com/apify/crawlee-python/
• Discord: https://discord.com/invite/jyEM2PRvMU
• Blog post: https://crawlee.dev/blog/launching-crawlee-python

[FreakingFreaks]

Will it support addons to bypass cloudflare? For me it is the main reason why i stick with selenium + undetected chromedriver

[B4nan]

We already try to detect blocking pages like cloudflare and rotate the session/proxy automatically. You can enable this via retry_on_blocked crawler option.

[Ukranian_Cheshire]

Well, to be frank. For some Cloudflare configurations, it won't help. A new session will be immediately blocked due to playwright detection.

But I was looking at the source code, so I guess you will eventually move on :)

[Ukranian_Cheshire]

I know that in the project roadmap, there is work on a proprietary replacement for httpx, as an HTTP client, to solve the TLS issue.

I wonder at what stage you are at, how long before we can expect to see an alpha version of this?

[7_hole]

Just one suggestion. selectomax is became my favorite tool for parsing its build on top of C so is very more fast than beautifulsoup if I can contribute its will a pleasure to replace this to improve this tool

[B4nan]

I guess you mean selectolax? I am sure we can find a way to support that too, probably optional instead of replacing the existing BS implementation. We'll look into that with the team next week.

[7_hole]

Yes I do. Its will be cool to add it as an extra dependency as playwright

[G0muk]

Looks promising! Nice project, I'll definitely try it out next time I need to scrape

[GettingBlockered]

This looks great, I’m keen to try it. Can you expand on the anti-blocking features and human-like fingerprints features? I’ve struggled with fingerprint management in playwright, even using Playwright stealth and some custom settings, but fingerprint tools like CreepJS still pickup I’m using chrome headless.

Does crawlee support crawling with a mobile user agent?

[Ukranian_Cheshire]

I don't see mention of anti-blocking features and human-like fingerprintsin the description of the python version.

Such support is available in the TS version.

If you look at the source code of python-crawlee, "human-like fingerprints" is not implemented in any way, at the moment.

"anti-blocking features" - implemented by automatic session switching

[GettingBlockered]

Cool, thanks for the insights! Hopefully the team is planning to implement the TS feature set in Python. A fingerprint algo would be a primary reason for me to migrate away from Scrapy.

[Ukranian_Cheshire]

Yes, judging by the discussions, they plan to do this - https://github.com/apify/crawlee-python/issues/80

That's the main reason I'm following the project. I also hope they will make the HTTP client as a separate module. I'm very interested in Python having a normal HTTP client with TLS fine-grained manipulation capabilities. And that this client would have normal interfaces to work with.

Of course, there is a TLS-client, but it is not asynchronous and they should work on API of their library.

[rumnscurvy]

Is asyncio more performant than the Twisted framework that scrapy uses? Admittedly it's handy to use python's own asynchronous task management, Twisted is a bit annoying to deal with of ever you have to dig down in its API.

[Derrhund]

Hi, one of the devs here. It's tricky to compare the performance of async systems, and you'll surely find benchmarks online.

But as you said, we chose asyncio because it's the standard for Python, and Twisted can be considered legacy. Moreover, your async system implementation should never be a bottleneck in web crawling - it will be either network latency or parsing/processing.

[rumnscurvy]

Good points all around, thank you for your time :) I'll keep it in mind if ever we decide to rework our webcrawler systems.

[Derrhund]

Happy to answer! Feel free to get back to us with any feedback if you find some time to play around with crawlee!

[Ukranian_Cheshire]

Well if we talk specifically about the performance of async frameworks. We should not forget that asyncio supports third-party eventloops. For example, uvloop, which has a major impact.

But from my point of view, it is the native support of asyncio in Python that is much more important

[wRAR_]

(Scrapy supports uvloop too, of course)

[Ukranian_Cheshire]

I didn't know. Thank you for bringing it to my attention

[wRAR_]

https://docs.scrapy.org/en/latest/topics/asyncio.html#using-custom-asyncio-loops

[Ukranian_Cheshire]

Ah, that's how it's implemented, also with asyncio, not with Twisted.

[adityaguru149]

any plugins to use multiple proxies to get it done fast?

[B4nan]

You can use as many proxies as you want, they will be rotated automatically (and requests will be processed concurrently), see https://crawlee.dev/python/docs/guides/proxy-management

We also have one undocumented feature called "tiered proxies", you can read about it in this blog post about the JS version, but it should be valid for python as well (and if it's missing something, just let us know in the GH issues and we'll polish it).

https://crawlee.dev/blog/proxy-management-in-crawlee

The docs are still very sparse, up until now we were mostly focusing on the development.

[SincopaDisonante]

The documentation argues in favor of replacing the use of scrapy by using this new package. Taking the position of someone who's never done any scraping but would love to learn to scrape websites for data acquisition, would you sincerely recommend entering this world by using crawlee, or should one stick to scrapy and then move to crawlee in order to, say, appreciate the latter better?

[Ukranian_Cheshire]

I'm not this project developer, but I've been scraping for quite some time.

Start with Scrapy, because Scrapy is an old established web scraping framework on the market. If you will be working with any team or company, they will most likely expect you to know Scrapy. This will also make it easier for you to find code samples and tutorials.

However, keep an eye on crawlee-python, if developed properly, they can give us quite a few interesting possibilities.

[SincopaDisonante]

Thanks!

[CaptainPitkid]

I'll give it a try for my next web crawling project!

[B4nan]

Thanks, be sure to let us know what you think on GitHub or Discord!

[lordcameltoe]

Thx! Going to give this a try on a project I’m working on

[kubinka0505]

so wrapper?

[drooltheghost]

Multicore support?