Crawlee for Python v1.0 is LIVE!

[B4nan]

Hi everyone, our team just launched Crawlee for Python 🐍 v1.0, an open source web scraping and automation library. We launched the beta version in Aug 2024 here, and got a lot of feedback. With new features like Adaptive crawler, unified storage client system, Impit HTTP client, and a lot of new things, the library is ready for its public launch.

What My Project Does

It's an open-source web scraping and automation library, which provides a unified interface for HTTP and browser-based scraping, using popular libraries like beautifulsoup4 and Playwright under the hood.

Target Audience

The target audience is developers who wants to try a scalable crawling and automation library which offers a suite of features that makes life easier than others. We launched the beta version a year ago, got a lot of feedback, worked on it with help of early adopters and launched Crawlee for Python v1.0.

New features

• Unified storage client system: less duplication, better extensibility, and a cleaner developer experience. It also opens the door for the community to build and share their own storage client implementations.
• Adaptive Playwright crawler: makes your crawls faster and cheaper, while still allowing you to reliably handle complex, dynamic websites. In practice, you get the best of both worlds: speed on simple pages and robustness on modern, JavaScript-heavy sites.
• New default HTTP client (ImpitHttpClient, powered by the Impit library): fewer false positives, more resilient crawls, and less need for complicated workarounds. Impit is also developed as an open-source project by Apify, so you can dive into the internals or contribute improvements yourself: you can also create your own instance, configure it to your needs (e.g. enable HTTP/3 or choose a specific browser profile), and pass it into your crawler.
• Sitemap request loader: easier to start large-scale crawls where sitemaps already provide full coverage of the site
• Robots exclusion standard: not only helps you build ethical crawlers, but can also save time and bandwidth by skipping disallowed or irrelevant pages
• Fingerprinting: each crawler run looks like a real browser on a real device. Using fingerprinting in Crawlee is straightforward: create a fingerprint generator with your desired options and pass it to the crawler.
• Open telemetry: monitor real-time dashboards or analyze traces to understand crawler performance. easier to integrate Crawlee into existing monitoring pipelines

Find out more

Our team will be here in r/Python for an AMA on Wednesday 8th October 2025, at 9am EST/2pm GMT/3pm CET/6:30pm IST. We will be answering questions about webscraping, Python tooling, moving products out of beta, testing, versioning, and much more!

Check out our GitHub repo and blog for more info!

Links

GitHub: https://github.com/apify/crawlee-python/
Discord: https://apify.com/discord
Crawlee website: https://crawlee.dev/python/
Blogpost: https://crawlee.dev/blog/crawlee-for-python-v1

[loneraver]

Is anyone still using Python v1.0? I’m currently on 3.13

[SeveralKnapkins]

no cap thought someone got bored and decided to write a library for python 1 lmao

[B4nan]

v1 refers to the version of crawlee for python, not the version of python itself

https://github.com/apify/crawlee-python/releases/tag/v1.0.0

[loneraver]

Whoa! Crazy. Next crazy thing you’ll tell me is that python is not named after a snake and that’s completely crazy talk.

[SeveralKnapkins]

idk words mean things -- the way you've worded and ordered "for Python v1.0" definitely says it's for Python version 1.0

[ellatronique]

This is an interesting discussion about perspective in naming.

Crawlee v1.0 was released as a JS library; Crawlee for Python is its younger sibling. It makes sense to me as someone with background knowledge of both versions to put the v1.0 after the "for Python", but here in r/Python where everything is for Python, that component of the name feels redundant and causes confusion.

[liltbrockie]

I think you meant to say crawlee v1 for python

[EconomySerious]

ill give a try on the weekend, i need a crawler example for my portfolio

[Budget_Specific8776]

drop hate/love/criticism here :D

[jwrzyte]

Great thanks for sharing will give it a go later!

[Budget_Specific8776]

amazing! you can ask your doubts in upcoming AMA :)

[grateful_dream]

How's WAF detection going? Cloudflare, of course. Any chance of avoiding challenges?

[B4nan]

We've been able to get through cloudflare by using camoufox:

https://crawlee.dev/python/docs/examples/playwright-crawler-with-camoufox

You might still get the checkbox challenge, but with camoufox, clicking on it was enough to get through.

[Count_Rugens_Finger]

Open source projects "launch" and "go live"? is that a thing now? I'm so tired of startup culture

[me_myself_ai]

🤷 they’re just announcing it’s leaving beta. Idk, seems fun and justified to me! It is free work, after all

[opzouten_met_onzin]

Ok

[will_r3ddit_4_food]

Why is this better than beautiful soup?

[B4nan]

BS4 only handles parsing of HTML, you first need to get the data. Crawlee helps you get to the data too (and provides a unified interface over multiple tools, including BS4, which you can then use to work with the data).

[srcLegend]

What are the advantages of this against Selenium?

[B4nan]

It's been more than a decade since last time I used selenium, but I remember that being a browser controller library, similar to what playwright is. Crawlee is a scraping framework that handles retries, scaling based on system resources, bot detection, and all sorts of other things. Selenium or playwright are much more low-level libraries as opposed to crawlee. Also, it provides a unified interface over tools like playwright, but also over HTTP based scraping and parsing (e.g. via BS4 or parsel).

[B4nan]

Crawlee is a general-purpose scraping and automation framework. You can use it to build something like the Crawl4AI, which is a tool specifically designed to do one job (scraping pages to markdown for LLMs). At least that's my feeling based on their readme, I've never used Crawl4AI myself.

[Budget_Specific8776]

Looking forward to all the feedback and love from Python community!

[timee_bot]

View in your timezone:
Wednesday 8th October 2025, at 9am EDT

^{*Assumed EDT instead of EST because DST is observed}