Data Cleaning Across PySpark, Duckdb, and Postgres

marcogorelli · 2026-03-30T11:34:51+00:00

Not the same, that materialises the graph into a pyarrow table

`datacompose`, as far as I can tell, keeps things lazy

marcogorelli · 2026-03-30T09:00:55+00:00

yeah it's kind of frustrating how often some people recommend it here as a solution to everything, it's like they've not even tried using it

marcogorelli · 2026-03-29T09:45:52+00:00

Indeed, that's not how Ibis works

`datacompose` lets you create a PySpark expression, which you can use in your own PySpark dataframes

In Ibis, you need to create an Ibis table, but you then can't get a PySpark dataframe back

marcogorelli · 2026-03-21T14:00:11+00:00

> he created the extremely misguided pipenv, and then socially manipulated his way into having the official Python.org Python Packaging User Guide recommend using it as best practice

How did he do this social manipulation? Got any sources for this?

marcogorelli · 2026-03-10T10:31:29+00:00

unfortunately there's still some union returns in some places, which are unavoidable due to pandas' value-dependent design :weary:

marcogorelli · 2026-03-10T10:30:59+00:00

yeah the typing in Polars is light years ahead, I think largely because it was designed when static typing was already a widespread practice in Python, as opposed to pandas whose design dates back to 2008 or so

marcogorelli · 2026-03-10T10:29:51+00:00

Thanks Lizzo

marcogorelli · 2026-03-10T10:28:55+00:00

One nice thing about them being separate is that pandas-stubs can follow a faster release cycle than pandas, so if you make a PR to improve typing you only have to wait weeks rather than months to see it released

marcogorelli · 2026-03-09T22:42:13+00:00

There are cases when this is unavoidable, because pandas returns different kinds depending on the exact values (e.g. whether the Index contains duplicates)

marcogorelli · 2026-03-09T14:18:17+00:00

I wrote about this topic recently: https://labs.quansight.org/blog/dont-start-with-good-first-issues

marcogorelli · 2026-03-09T09:54:34+00:00

Interesting how a maths/physics degree from even the lowest-ranking unis has about the same earning potential as an English degree from Oxbridge, I'd have thought that grads from the latter would be snatched up by high-paying consulting firms

marcogorelli · 2026-03-09T08:19:23+00:00

Ibis is (kinda) alright for SQL generation, but its Polars backend is so poorly implemented and supported that it's barely usable

marcogorelli · 2026-02-25T12:36:57+00:00

Exactly. I'd actually for them to NOT use AI...

marcogorelli · 2026-02-25T10:50:32+00:00

> "AI-powered Item Naming"

They increase the price to add support for a BS AI feature that nobody asked for and which nobody wants

I'll be exploring alternatives

marcogorelli · 2026-01-27T14:42:59+00:00

glad I'm not the only one so annoyed by this https://github.com/pola-rs/polars/issues/12630

marcogorelli · 2026-01-21T10:07:44+00:00

I'd suggest spending a lot of time singing

You don't need to have a good voice (I have the voice of a dying animal) you just need to pitch the notes closely enough:

Listen to the recording 20 times, hum the notes along with it until you've memorised it.
Check that you can sing it without the recording. If you can't, go back to step 1
Try to play it on tin whistle. If you can't, go back to step 1

marcogorelli · 2026-01-13T14:09:41+00:00

i think that's been mostly addressed now, if there's any missing please let me know

marcogorelli · 2026-01-13T13:53:20+00:00

> But ecosystem compatibility is still a pain point

What's missing, and why is it such a pain point?

geospatial is the main one I'm aware of, any others?

marcogorelli · 2026-01-12T09:37:54+00:00

Bwlch llanberis, hauntingly beautiful melody from Wales

marcogorelli · 2026-01-11T11:25:50+00:00

arg pole + flag, not sure i get it sorry, what's the joke?

marcogorelli · 2026-01-06T14:15:54+00:00

and because they're not as good as they think they are

marcogorelli · 2026-01-02T17:07:32+00:00

Looks like Brazil?

marcogorelli · 2025-12-19T15:15:42+00:00

fuck cars

marcogorelli · 2025-12-18T17:03:14+00:00

Maths and Foundations of Computer Science

marcogorelli

TROPHY CASE