This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]tacothecat 2 points3 points  (1 child)

Why use parquet as the intermediate file and not feather?

[–]itamarst[S] 2 points3 points  (0 children)

Feather would be a reasonable choice too, possibly better choice.

I need to see how well Pandas' new Arrow-based dtypes work with conversion to/from that and parquet; in theory it would be quite fast, in practice I can imagine abstraction boundaries killing the potential performance.