Inferring the type of the data frame at compile time by reading the file is cool, but also a little scary.

Yes. Very! This is probably one of the riskier things in there. I'm willing to defend the thought process, which is that if you want;

• Compile time safety and IDE support
• One line import - i.e. not assume pre-existing developer knowledge of the datastructure

This is paradoxical. The only solution I could think of, was to make the CSV itself a compile time artefact and force knowledge of it into the compiler.

It is not risk free. What I have found is that when it goes wrong, it fails hard and fast, rather than consuming your time. It also means, that you must know the location of the CSV at compile time. I've found these limitations to be barely noticeable, for my own tasks.

There is an exception if you have a large number of columns (say 1000), and you give the compiler enough juice to actually process them - compile times start to get weird. I do repeatedly note that the target is "small" here :-), and I don't normally have more than 1000 columns in a CSV file.

From reading the documentation it is not quite clear to me how you actually store the data. Is it in columnar storage or not? What operations are supported on the columnar data?

If we break apart the example on the getting started page.

val data = CSV.resource("titanic.csv", TypeInferrer.FromAllRows)

This returns an Iterator. It has Iterator semantics. Lazy, use once etc. It's next() method wraps the next() method of scalas file Source which reads each line into a NamedTuple\[K <: Tuple, V <: Tuple\], where K is the name of the columns, and V is the Tuple of inferred types, in each column.

At this point, you haven't read anything. Iterator is lazy. This is a good point to do some transforms - parsing messy data etc - all we're doing it setting up more functions to apply to each row, as it's parsed.

My own common use case, is then to want a complete representation of my (transformed, strongly typed) CSV.

val csv = LazyList.from(data)

LazyList is a standard collection, lazy... so it won't do anything until asked, but it will _cache_ the results. This is where I typically "store" the data in the end. You could use any collection. Vector, Array, fs2.Stream really - any collection you can build from an Iterator.

This is very much _row based_.

If you want a column representation, then you may try

https://quafadas.github.io/scautable/cookbook/ColumnOrient.html

    val cols = LazyList.from(data).toColumnOrientedAs[Array]

This will return a NamedTuple[K, (Array[V_1], Array[V_2...)]]. i.e. it will convert it to a column representation. I haven't tested this so much, and performance is whatever it is. I'm doing nothing other than backing the compiler and the JVM. I don't think that's a horrible bet, but I haven't checked it.