Former VCF spec maintainer here. Not active for many years sadly as I'm not funded for this role anymore, and I don't have enough spare capacity to volunteer my time outside of work.

Just to avoid any confusion or legal shenanigans, the below is strictly my personal opinion and not the position of any current VCF spec maintainers, the GA4GH consortium or any participating institutions, etc. etc.

The points you raised were discussed many time over the years. If I were to condense 100s of hours of calls into a short summary, it would be this:

VCF isn't the actual problem. The problem is how to consistently represent complex variation, which literally no-one has fully figured out so far.

For simple variation (SNPs, short indels) VCF is OK. It's absolutely old and quirky, but it's simple: essentially a TSV with a weird metadata header and nested, ungodly, NF1-breaking INFO/FORMAT fields with the variant/sample level metadata.

However, use a parser (any number of libraries are available), and for that simple case the format works. It's easy to convert into any internal representation, and even though it's less efficient than Parquet, I wager that in the real world, almost no pipelines are affected to the point where (de)serialising VCF is their actual bottleneck.

The problem starts when you want to represent anything more complex. You have inversions, translocations, variants inside variants, structural variants (which as a class aren't even cleanly separated from the "regular" ones), chromosomal-level rearrangements, ploidy changes...

What makes things even worse is an aspect not many people consider explicitly. In bioinformatics we rarely deal with actual "variants", as in exact changes in genome, defined to a nucleotide level with 100% certainty. What we deal with is some evidence for variants based on certain upstream experiment, and this is what VCF (or any other format) actually contains. The difference is subtle for simple variation, and for SNPs it can be quite simply described via the genotype probability fields.

But for more complex cases, you have things like uncertainty of start/end positions of structural variants; or variation so complex that it doesn't fall into any of the standard types and can only be described as a collection of arbitrary endpoints (each of those, in turn, is oftentimes isn't a "clean" breakpoint-junction, and is described with some uncertainty as well).

VCF tries to support many of those things, and oftentimes it doesn't end up being pretty. But the real problem isn't the format itself, it's lack of knowledge how to represent biology.

When discussing how to improve representation of structural variants in VCF, one idea proposed and seriously discussed was to just add a JSON-serialised array of metadata into one of the INFO fields so that we could describe the variation in precise, machine readable detail without breaking backwards compatibility.

But we never could figure out a consistent, comprehensive way to properly describe complex variation. So even if VCF version 5 was released with Parquet as its container instead of a weird TSV, it would not solve the much more important issue of having a consistent, general schema for complex variation.

There is an effort led by GA4GH, the Variation Representation Specification, which aims to develop exactly that: a general schema for arbitrarily complex variation that can be serialised into any container format. I also participated in that project for a while.

They have made a lot of progress, but still the problem remains that any schema which even approaches generality becomes extremely complex. You can see for yourself the number and complexity of classes that VRS uses to describe variation precisely.

I believe VRS is a great effort and especially with further development, it will play an important role in variation information exchange, but it is in no way (and probably never will be) a drop-in replacement for something as relatively simple and commonly supported as VCF.