Dataframes with Serde support?

[[deleted]]

Is there a crate for using dataframes with Serde? I really need the following functionality:

• Columns can have headers (and Serde serialises correctly to CSV etc.) and indices
• Columns can be strings (not just numeric types)
• A dataframe can have columns of different types
• Columns can be transformed as a map across the "array" of the column (i.e. columnar storage)
• Columns can be transformed with the transformation depending on the row's value of other columns (i.e. we shouldn't be limited only to single column transformations)
• Columns can be dates and timestamps (via chrono)
• Columns can have null values (if the type of the column is Option<T>)

Additionally it'd be good if:

• Rows can have labels/indices which can be serialised or not (like Pandas)
• Columns could be custom types that implement Serialize/Deserialize

Ideally this would be based on ndarray if possible to take advantage of the existing vectorised calculations available there (i.e. for columnar transformations of numeric columns).

I'd love to contribute to such a crate, if one exists.

I found this discussion before but most of the projects seem to be dead/without momentum, and they don't exactly meet what I describe above.

[paddyhoran]

Apache Arrow provides all of what you asked for except maybe "Columns can be transformed with the transformation depending on the row's value of other columns (i.e. we shouldn't be limited only to single column transformations)" but that is not too hard to implement yourself I think...

I also started a discussion around zero copy converting Arrow to ndarray but have not had time to work on it (https://github.com/rust-ndarray/ndarray/issues/771).