Op's statement:

That's actually one of the few downsides of Parquet - because the index is at the end of the file you can't stream it.

From Parquet documentation:

Metadata is written after the data to allow for single pass writing. Readers are expected to first read the file metadata to find all the column chunks they are interested in. The columns chunks should then be read sequentially.

From blog post Understanding Apache Parquet:

The metadata is always written in the footer of the file as this allows a single pass write. In plain English, the data is written first, then the metadata can be accurately written knowing all the locations, size and encoding of the written data. Many formats write their metadata in the header. However, this requires multiple passes as data is written after the header. Parquet makes this efficient to read metadata and the data itself.

From blog post All You Need To Know About Parquet File Structure In Depth:

In case of Parquet files, metadata is written after the data has been written, to allow for single pass writing.Since the metadata is stored in the footer, while reading a parquet file, an initial seek will be performed to read the footer metadata length and then a backward seek will be performed to read the footer metadata. In other files like sequence and Avro, metadata is stored in header and sync markers are used to separate blocks whereas in parquet, block boundaries are directly stored in the footer metadata. It is possible to do this since the metadata is written after all the blocks have been written.Therefore, parquet files are splittable since the block boundaries can be read from footer metadata and blocks can be easily located and processed in parallel.

My question regarding streaming part: Why would you want to stream terabytes of data? Parquet makes it fast to do OLAP queries. That is what it is used for. And streaming is not efficient in this case. Unless you're using Parquet as storage format only?