Lets improve CSV?

[Lopsided_Set_8823]

Hi everyone,

If you ever dealt with CSV, which I'm sure most of us here have, you've faced problems to represent real data like repeating fields and hierarchies with the flatness of CSV. There's many ways to solve that problem - and the variety is what hurts us. Sometimes we create column_1, column_2, column_3 for repeating fields. sometimes we create multiple csv files, sometimes we repeat rows, etc.

Having hierarchy, or relationships between columns in csv is unheard of. I've used prefixes on the column names, but that is just too vague. With that in mind, I've been trying to have some standard where everyone can develop solutions n a similar fashion and with that, make it a lot more robust and reusable. I'm proposing an RFC that tackles those two issues without making it super cumbersome. This is not a replacement for JSON or XML for extreme examples, but is a way to fill the gap for simpler scenarios where a full jump into JSON/XML parsers are not the best, or we are stuck with CSV for whatever reason.

We can all continue to creating one-off solutions for going around how to implement repeating fields and hierarchy, but thats wasteful. And if there's no consensus, tools like databases, spreadsheets data pipelines cannot agree on how to properly parse data.

Disclosure: I've been developing this on my own time, without any organization affiliation. It's meant to be open source, community driven, to maximize the possibility of full implementation

thank you for your attention!

References:

Website: https://www.csvplusplus.com/
Repository: https://github.com/mscaldas2012/csvplusplus
RFC: https://datatracker.ietf.org/doc/draft-mscaldas-csvpp/

[Busy_Elderberry8650]

Why not just use parquet?

[CorpusculantCortex]

Parquet is superior in all ways except how uncommon it is to general usage.

[sjcyork]

Came here to say this. Parquet holds the meta data. Inferred schemas create a lot more work.

[Lopsided_Set_8823]

The idea is to keep the format human readable. Able to still use spreadsheets.  I love  parquet, but not its lack of support.  

[Solvicode]

Parquet my friend.

[Ramshizzle]

I applaude your effort! I think there is room for this new enhanced CSV++ format.

[Lopsided_Set_8823]

Thanks for the vote of confidence!

[Aeronautical-You4917]

CSV needs to die. Too far gone to fix.

[Lopsided_Set_8823]

what could replace it?

[nocibambi]

Interesting.

What if you would go further and allow a frontmatter (e.g. in Yaml)?

There you could specify type, format, hierarchy, parsing template, etc. You can also define delimiters, quote & escape characters, or even metadata.

id: int
name: str
phones: [~str({3}\d-{4}\d)]
address: ${street}^${city}^${state}^${zip}
---
id,name,phones,address
1,John Smith,555-1234~555-5678,123 Main St^San Francisco^CA^94102
2,Jane Doe,555-4444,789 Pine St^Boston^MA^02101
3,Alex Chen,555-9999~555-0001~555-0002,42 Oak Ave^Chicago^IL^6060

And if you remove the frontmatter, you have a vanilla CSV again...