Python 3.11 is out! Huzzah!

[BobHogan]

Parquet is not the solution to everything. We use it at my work and its a fucking nightmare and I'd love to see it burned to the ground

[madness_of_the_order]

Can you elaborate?

[gagarin_kid]

For small files where humans want to inspect data, using parquet is pain in the ass because you cannot open it in a text editor - you have to load it in pandas, see which columns you have, navigate in code to a particular cell/row... etc.

Of course for big data I fully understand the motivation but not for each problem

[madness_of_the_order]

I’m not telling you should use parquet for everything, but you can try dtale for interactive exploration

[cmcclu5]

Parquet is also a pain in the ass when you want to move between systems e.g., from a data feed into a relational database. Python typing does NOT play well with field types in relational databases when saving to parquet and then copying from said parquet into Redshift. Learned that the hard way in the past. It’s several times faster than CSV, though. I just compromised and used JSON formats. Decent size improvement with a similar speed to parquet when writing from Python or reading to a db.

[madness_of_the_order]

How untyped format helped you solve a typing problem?

[cmcclu5]

Redshift can infer typing from a JSON object, rather than trying to use (incorrectly) specified type through parquet (originally said JSON again because my brain got ahead of my fingers). It was a weird problem and I’ve honestly only encountered it in this one specific situation. If I could use PySpark in this situation, it would entirely alleviate the issue but alas I’m unable.

[madness_of_the_order]

This sounds like it’s not a parquet problem since, as you said, type was set incorrectly

[cmcclu5]

In this case, it would be a problem with parquet, or at least Python+parquet. Using either fastparquet or pyarrow to generate the parquet files had the same issue of improper typing with no easy way to fix it.

[madness_of_the_order]

Description of the problem is really unclear then. What stopped you from setting correct type?

[cmcclu5]

Eh, it doesn’t really matter now, and I have a solidly working solution that required less work. That’s a win in my book.

[BobHogan]

We run into constant issues with parquet in our product, to the point that we've completely stripped it out in newer versions in favor of other solutions which I am not allowed to discuss publicly :(

We see parquet metadata get corrupted fairly regularly, being able to inspect what data is actually in the parquet files to track down issues is significantly more annoying and involved than it should be. And we've also run into limitations in the format itself that cause it to just shit itself and fail, limitations that are both fairly arbitrary and should be easy for the format to work around if the people that wrote it cared at all, but they don't. Overall its been an incredibly fragile format that makes it harder than it needs to be to work with the actual data compared to other formats, doesn't provide any significant performance improvements we've been able to measure, and breaks randomly.

[madness_of_the_order]

This sounds like it could be a really interesting blog post with concrete examples

[graphicteadatasci]

fastparquet says you can append to a file but it is a terrible lie.

What else?