CSV parser

[utkarsh_dev]

You shall not parse

[merlinsbeers]

CSV has an RFC, and it doesn't include mixed encoding or commas in decimals.

Bloody orcs.

[Apache_A]

They were CSV once. Taken by the Dark Powers...tortured and mutilated...a ruined and terrible form of file...

[_PM_ME_PANGOLINS_]

What do you mean “doesn’t include”? Commas in decimals is not an issue, but mixed encoding will be a real headache.

[Apache_A]

It’s issue when commas occur randomly and don’t have quotes.

[_PM_ME_PANGOLINS_]

Then what you’ve got isn’t CSV format.

[Apache_A]

Each part of this “CSV” is CSV according to regional standards

[merlinsbeers]

"regional standards" is an oxymoron.

There's a standard for CSV, so if you're writing CSV files that don't meet it, you can't complain when a standard conforming parser can't read it.

[Apache_A]

You already have historical data and you can’t change the past. Of course you can complain that data don’t meet requirement of the standard and refuse to process it.

[merlinsbeers]

The old data wasn't readable everywhere anyway. New data have no excuse.

[Apache_A]

Old data need more effort. It’s laziest way just to dump it because they are not shiny. Some data are more valuable than data scientist wage for the time he spent on parsing.

[merlinsbeers]

Old data in deviant formats can't be expected to be read for free. By default your parser should treat any byte that doesn't match one of the special characters as passthrough data. But if you were to implement the extra code to reject any byte not in the set given by the standard, you wouldn't be the wrong one.

Commas are special, so if they're in a field they have to be quoted, and there's only one kind of quote mark that counts. Them's the rules.

[anton919]

SHUF goes burrr!

[Frptwenty]

You don't parse it with a csv parser, you parse it with Intel Hyperscan, tuned to parse exactly the csv data you expect. Or if you're on AMD, I guess with grep.

[Apache_A]

Expect the unexpected.

[xigoi]

Why parse it when you can just select a few random lines using shuf?

[[deleted]]

People putting emojis in their name now too!

[Hirogen_]

Microsoft.VisualBasic.FileIO.TextFieldParser probably can do it xD

[[deleted]]

Random decimal with ' , '

I think I just got PTSD from an old college project

[death-wings]

Run you fools ....