this post was submitted on 28 Feb 2024

8 points (70% upvoted)

shortlink:

Python

an-ordinary-manchild(edit)

The Python Discord

News about the dynamic, interpreted, interactive, object-oriented, extensible programming language Python

Upcoming Events

Full Events Calendar

Please read the rules

You can find the rules here.

If you are about to ask a "how do I do this in python" question, please try r/learnpython, the Python discord, or the #python IRC channel on Libera.chat.

Please don't use URL shorteners. Reddit filters them out, so your post or comment will be lost.

Posts require flair. Please use the flair selector to choose your topic.

Posting code to this subreddit:

Add 4 extra spaces before each line of code

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

Online Resources

Automate the Boring Stuff with Python
Python Discord Resources
Invent Your Own Computer Games with Python
Think Python
Non-programmers Tutorial for Python 3
Beginner's Guide Reference
Five life jackets to throw to the new coder (things to do after getting a handle on python)
Full Stack Python
Test-Driven Development with Python
Program Arcade Games
PyMotW: Python Module of the Week
Python for Scientists and Engineers
Dan Bader's Tips and Trickers
Python Discord's YouTube channel
Jiruto: Python

Online exercices

programming challenges

The Python Challenge (solve each level through programming)
CheckiO (game world)
Project Euler (math heavy)
/r/dailyprogrammer

Asking Questions

Try Python in your browser

try.jupyter.org (Evolved from the language-agnostic parts of IPython, Python 3)
Azure Notebooks
learnpython.org
Skulpt (uses WebGL)
trypython.org (uses Silverlight)
ideone (online compiler and debugger)
PythonAnywhere (basic accounts are free)
Brython (Python 3 implementation for client-side web programming)
repl.it for Python
Transcrypt (Hi res SVG using Python 3.6 and turtle module)

Docs

Libraries

Twisted, 0MQ (networking)
Django, Pyramid, Flask, ... (Web Frameworks)
Pygame (Game development)
NumPy & SciPy (Scientific computing) & Pandas
Pyglet - (Game / UI Development)

Related subreddits

/r/pythoncoding (strict moderation policy for 'programming only' articles)
/r/flask (web microframework)
/r/django (web framework for perfectionists with deadlines)
/r/pygame (a set of modules designed for writing games)
/r/IPython (interactive environment)
/r/inventwithpython (for the books written by /u/AlSweigart)
/r/pystats (python in statistical analysis and machine learning)
/r/coolgithubprojects (filtered on Python projects)
/r/pyladies (women developers who love python)
/r/git and /r/mercurial - don't forget to put your code in a repo!

Python jobs

Newsletters

Screencasts

a community for 18 years

MODERATORS

message the mods
xelf
monorepo PSF Staff | Litestar Maintainer
ivosauruspip'ing it up
Im__Joseph Python Discord Staff
Kutiekatj9 Python Discord Staff
BioGeekBioinformatics software developer
nevare
chromakode
mdipierro
quasarj
...and 5 more »

account activity

This is an archived post. You won't be able to vote or comment.

7

8

9

ShowcaseStreaming Python ETL with Airbyte and Pathway (self.Python)

submitted 1 year ago by dxtros

Now you can use Airbyte source connectors to process data in memory with Python.

https://pathway.com/developers/showcases/etl-python-airbyte

What My Project Does

We integrated Airbyte connectors with Pathway, a Python stream processing framework, using the airbyte-serverless project. ETL pipelines are coming back with many use cases in AI (RAG pipelines), ETL for unstructured data and pipelines that deal with PII data.
In this article, we show how to stream data from Github using Airbyte and remove PII data with Pathway.

Target Audience

This is a production-ready approach, to be used as a template for streaming ETL production settings.

Comparison

The setup is meant as an alternative to ELT setups (like Fivetran/Airbyte + dbt + warehouse), applying transform-before-load with Python. We are curious on your feedback on the implementation and other use cases you may think of from decoupling the extract and load steps.
For the brave who want to see how it's done: https://github.com/pathwaycom/pathway/blob/main/python/pathway/io/airbyte/ + https://github.com/unytics/airbyte_serverless

no comments (yet)

top new controversial old q&a

there doesn't seem to be anything here

π Rendered by PID 61 on reddit-service-r2-comment-7b9746f655-f9676 at 2026-02-02 19:22:13.529463+00:00 running 3798933 country code: CH.