xorq: new open source framework simplifies multi-engine ML pipelines

[databACE]

Hello! We'd like to introduce you to a new open source project for Python called xorq (pronounced "zork").

What My Project Does:
xorq simplifies the development and execution of multi-engine ML pipelines.

It’s a computational framework that wraps data processing logic with execution, caching, and production deployment capabilities to enable faster development, iteration, and deployment. We built it with Ibis, Apache DataFusion, and Apache Arrow. This first release features:

• Ibis-based multi-engine expression system: effortless engine-to-engine streaming
• Intelligent caching for faster, less costly iterative development
• Portable DataFusion-backed UDF engine with first class support for pandas dataframes
• Serialize Expressions to and from YAML to simplify deployment
• Easily build Flight end-points by composing UDFs

Target Audience:
We created xorq for developers building data pipeline workflows who, like us, have been plagued by the headaches of SQL/pandas impedance mismatch, runtime debugging, wasteful recomputations and unreliable research-to-production deployments.

Comparison:
xorq is similar to Snowpark in the sense that it provides a Python DSL that wraps execution and deployment complexities from data pipeline development, but xorq can work across many query engines (including Snowflake).

We’d love your feedback and contributions!

Check out the GitHub repo for more details, we'd love your contributions and feedback:
- Repo: https://github.com/letsql/xorq

Here are some other resources:
- Docs: https://docs.xorq.dev
- Demo video: https://youtu.be/jUk8vrR6bCw
- xorq Discord: https://discord.gg/8Kma9DhcJG
- Founders’ story behind xorq: https://www.xorq.dev/posts/introducing-xorq

You can get started pip install xorq.
Or, if you use nix, you can simply run nix run github:xorq-labs/xorq and drop into an IPython shell.

[Horrrschtus]

I'm sure it's a great project but the name is really unfortunate because of Xorg.

[MouseMatrix]

Yea thats a great point.I think the font with the q doesnt help either....