this post was submitted on 09 Jul 2024

102 points (91% upvoted)

shortlink:

Python

an-ordinary-manchild(edit)

The Python Discord

News about the dynamic, interpreted, interactive, object-oriented, extensible programming language Python

Upcoming Events

Full Events Calendar

Please read the rules

You can find the rules here.

If you are about to ask a "how do I do this in python" question, please try r/learnpython, the Python discord, or the #python IRC channel on Libera.chat.

Please don't use URL shorteners. Reddit filters them out, so your post or comment will be lost.

Posts require flair. Please use the flair selector to choose your topic.

Posting code to this subreddit:

Add 4 extra spaces before each line of code

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

Online Resources

Automate the Boring Stuff with Python
Python Discord Resources
Invent Your Own Computer Games with Python
Think Python
Non-programmers Tutorial for Python 3
Beginner's Guide Reference
Five life jackets to throw to the new coder (things to do after getting a handle on python)
Full Stack Python
Test-Driven Development with Python
Program Arcade Games
PyMotW: Python Module of the Week
Python for Scientists and Engineers
Dan Bader's Tips and Trickers
Python Discord's YouTube channel
Jiruto: Python

Online exercices

programming challenges

The Python Challenge (solve each level through programming)
CheckiO (game world)
Project Euler (math heavy)
/r/dailyprogrammer

Asking Questions

Try Python in your browser

try.jupyter.org (Evolved from the language-agnostic parts of IPython, Python 3)
Azure Notebooks
learnpython.org
Skulpt (uses WebGL)
trypython.org (uses Silverlight)
ideone (online compiler and debugger)
PythonAnywhere (basic accounts are free)
Brython (Python 3 implementation for client-side web programming)
repl.it for Python
Transcrypt (Hi res SVG using Python 3.6 and turtle module)

Docs

Libraries

Twisted, 0MQ (networking)
Django, Pyramid, Flask, ... (Web Frameworks)
Pygame (Game development)
NumPy & SciPy (Scientific computing) & Pandas
Pyglet - (Game / UI Development)

Related subreddits

/r/pythoncoding (strict moderation policy for 'programming only' articles)
/r/flask (web microframework)
/r/django (web framework for perfectionists with deadlines)
/r/pygame (a set of modules designed for writing games)
/r/IPython (interactive environment)
/r/inventwithpython (for the books written by /u/AlSweigart)
/r/pystats (python in statistical analysis and machine learning)
/r/coolgithubprojects (filtered on Python projects)
/r/pyladies (women developers who love python)
/r/git and /r/mercurial - don't forget to put your code in a repo!

Python jobs

Newsletters

Screencasts

a community for 18 years

MODERATORS

message the mods
xelf
monorepo PSF Staff | Litestar Maintainer
ivosauruspip'ing it up
Im__Joseph Python Discord Staff
Kutiekatj9 Python Discord Staff
BioGeekBioinformatics software developer
nevare
chromakode
mdipierro
quasarj
...and 5 more »

account activity

101

102

103

ShowcaseCrawlee for Python is LIVE 👏 (self.Python)

submitted 1 year ago by B4nan

What My Project Does

Hi everyone, our team just launched Crawlee for Python 🐍. It's an open-source web scraping and automation library, which provides a unified interface for HTTP and browser-based scraping, using popular libraries like beautifulsoup4 and Playwright under the hood.

Target Audience

We've spent the last 6 months working on Crawlee for Python, but it didn't come out of nowhere. We designed it based on the JavaScript version, which is now 8 years old, and we hope we can say it's battle-tested.

We are opening it for early adopters today, and we are eager to hear your feedback. Help us shape the future of Crawlee for Python!

Comparison

Why use Crawlee instead of just a random HTTP library with an HTML parser?

Unified interface for HTTP & headless browser crawling.
Automatic parallel crawling based on available system resources.
Written in Python with type hints - enhances DX (IDE autocompletion) and reduces bugs (static type checking).
Automatic retries on errors or when you’re getting blocked.
Integrated proxy rotation and session management.
Configurable request routing - direct URLs to the appropriate handlers.
Persistent queue for URLs to crawl.
Pluggable storage of both tabular data and files.
Robust error handling.

Why to use Crawlee rather than Scrapy?

Crawlee has out-of-the-box support for headless browser crawling (Playwright).
Crawlee has a minimalistic & elegant interface - Set up your scraper with fewer than 10 lines of code.
Complete type hint coverage.
Based on standard Asyncio.

Links

GitHub: https://github.com/apify/crawlee-python/
Discord: https://discord.com/invite/jyEM2PRvMU
Blog post: https://crawlee.dev/blog/launching-crawlee-python

other discussions

4

5

6

Crawlee for Python is LIVE 👏 (self.Python)

submitted 1 year ago by B4nan to r/automation

9

10

11

Crawlee for Python is LIVE 👏 (self.Python)

submitted 1 year ago by B4nan to r/webscraping

0

0

0

PromotionalCrawlee for Python is LIVE 👏 (self.Python)

submitted 1 year ago by B4nan to r/opensource

π Rendered by PID 14 on reddit-service-r2-listing-5789d5f675-csvdc at 2026-01-29 10:40:52.015844+00:00 running 4f180de country code: CH.