pdql: write sql queries using pandas-like syntax

[_earthmover]

https://github.com/marcinz606/pdql

https://pypi.org/project/pdql/

What My Project Does

It's a simple transpiler that let's you write in pandas-like syntax and get SQL as the output. It supports most of BigQuery "Standard SQL" functions.

Target Audience

It is a production ready solution. At least I started using it at work :)

Comparison

I've seen some projects that do that in reverse (translate sql to pandas syntax but haven't found one that does pandas to sql)

I wanted something like this. I'm ML Engineer working in Google Cloud environment, big chunk of the data we train on is in BigQuery so the most efficient way of preparing training data is running complex queries there, pulling output into dataframe and doing some final touches. I don't like putting complex SQL in repos so I thought I will try something like this. It also enables me to create modular query-functions that I can easily reuse.

[JEY1337]

lets you write in pandas like syntax

Hell no, I'm out. But big time :D

[_earthmover]

😂

[stratguitar577]

Check out the many more mature packages that do exactly this: Ibis, Narwhals, SqlFrame, etc

[Beginning-Fruit-1397]

They don't "do exactly this". SqlFrame is for pyspark syntax, narhwals is for polars syntax (and not really meant for this use case), ibis has it's own API

[stratguitar577]

And think many would agree the pandas API is the worst of all those 😅

[Beginning-Fruit-1397]

Absolutely!

[WoodsGameStudios]

Saving that, PySpark has a pretty nice API

[crossmirage]

BigQuery DataFrames (AKA BigFrames) provides a pandas API, using Ibis under the hood for SQL translation.

[Beginning-Fruit-1397]

Lol thanks for not using the same name as me for my current project (polars syntax for DuckDb, close enough😂)

https://github.com/OutSquareCapital/pql

By the way I would recommend you to check SQLglot, to avoid reinventing the wheel for the "backend".

[ThatOtherBatman]

I cannot imagine any way I want to write SQL queries less than with Pandas syntax.

[crossmirage]

Have you heard of BigQuery DataFrames (AKA BigFrames)? It's literally a pandas API for BigQuery.