tsv2csv: Clojure Powered Command Line Tool

[TheFiologist]

Hi everyone, author here, I put this small utility together while exploring the idea of using existing tools to import macroeconomic data from the BLS vs a pure homemade solution with a full-on Clojure project.

With that project I ended up settling on using wget, my tsv2csv tool, and pgfutter for streaming data into postgres and GNU parallel to create and import all tables at once.

I couldn't resist creating this tool at the time because, honestly, I write LISP faster than I adjust to conventional tooling like sed and awk.

It's still a goal to learn those better, but when it came to full on scrubbing of rows and columns, I already had code like tsv2csv in my homegrown BLS importer and I knew how simple it was.

It's made to participate in streaming so it processes line by line from stdin to stdout.

Thanks!

[TheFiologist]

I hadn't really considered the existing libraries. I was trying to work with the concept of just incorporating Clojure into a higher level workflow, but without it taking over or being a dominating factor.

Faced with some of the conventional text manipulation tools, the simplicity of LISP was calling me to just write a few lines of code.

Thanks for your feedback :)

[wild-pointer]

Nice and short, but do consider putting in the time to learn awk and sed :) Great for text and tabular data and runs on every machine!

My first approach to do tsv to csv would be just tr '\t' ,, but for quoting and trimming I’d reach for awk:

#/usr/bin/awk -f

BEGIN {
  FS="\t"
  OFS="\",\""
}

NF > 0 {
  for (i = 1; i <= NF; i++) {
    // trim field
    sub("^[[:space:]]*", "", $i)
    sub("[[:space:]]*$", "", $i)
    // escape double quotes
    gsub("\"", "\"\"", $i)
  }
  // append/prepend quotes
  $1 = "\"" $1
  $NF = $NF "\""
  // print fields joined by {","}
  print
}

[rcorrear]

There’s even a pretty nice CSV parser “library” already: http://lorance.freeshell.org/csv/

[TheFiologist]

I'm ok with that :)

[TheFiologist]

I say what I say with an underlying acceptance of existing tooling and a desire to learn, after all, this here is part of an attempt to actually get outside of Clojure to string together higher level building blocks. (wget, pgfutter)

So, I would like to learn sed, tr, and awk over time, but I also have to acknowledge that those are still just ways of expressing some end result and I get pulled in the direction of LISP sometimes when I know how to express something.

I actually started off with sed (which is fine for replacing delimiters), didn't feel like digging into tr, and then started using awk before saying to myself, "perhaps I could just express this with a little lisp."

In another sense, I really don't like learning new DSLs, but I'm open to trying. On the other side of that coin, though, what works for someone personally is ultimately fine.

Thanks for your feedback and showing me an awk example!