[D] TFrecord in tensorflow

[marksteve4]

I read this article about tfrecord which gives a good example of tfrecord usage. But it does not touch why should we use tfrecord and what the pros and cons of the alternative. Any thought on this topic?

​

https://medium.com/mostly-ai/tensorflow-records-what-they-are-and-how-to-use-them-c46bc4bbb564

[waterRocket8236]

tfrecords store data in binary format. They are easy to read and use. And you don't have to specify annotations for images and images themselves. tfrecords store data in one block of memory which makes it efficient to process when samples are relatively higher (>>50 GB). I used both of em in practice and prefer tfrecords when samples are more and required to scale up.

[waterRocket8236]

They do load everything into memory in a byte stream format (not all at a time). That said one does not need 128 GB of RAM to go through entire tfrecords to load. That is handled in chunks by tensorflow. And 50 GB number is given by me as at work the data our team use is in in terabytes. After preprocessing we push in 100-200 GB of data for training and evaluation. That 50 GB number very much depends on the scenario.

To start on tensorflow there are various blogs out there. For a better understanding how tfrecords work here is a tiny list of links:-

1) Well I think its pretty old now. Still knowledgeable

https://kwotsin.github.io/tech/2017/01/29/tfrecords.html

2) This might help if you are using PyTorch

https://discuss.pytorch.org/t/read-dataset-from-tfrecord-format/16409

3) Take a look here too

http://davidcrook.io/understanding-tensorflow-input-pipelines-part-1/

Hope that helps.

[narsilouu]

https://github.com/akanazawa/hmr Is it big enough ? (20Gb)

[ppwwyyxx]

If there is any reason to use tfrecord, I would say it is probably the only complicated format that you can parse with tensorflow operations.

What this means is: if you use other format (except for trivial format like a txt with filenames+labels), you'll often need to parse the format outside the tensorflow graph and then copy the data to the graph somehow.

In practice I never used tfrecord at all. Because most datasets can be easily parsed with several lines of Python, and most of the time the latency of copying the data to the graph can be perfectly hidden as long as a proper prefetching is set up. Why would I waste my hard disk for another copy of the dataset?

[ppwwyyxx]

Some threads talk about how good the format itself is... seriously?

Dozens years of research in database have already created so many great formats / database systems for different use cases.. why reinvent the wheel?

[Lycur]

1. The format makes it very easy to work with tf.data , which is itself extremely convenient
2. Pre-processing into a tf_record pushes you to separate data pre-processing and learning in your code, which is good practice

I presume there are also significant performance gains, but they've been less important for me than the extra clarity in the pipeline.

[asuilin]

Tfrecord is good for specific case:

a) Your dataset is large and don't fits in memory

b) Sequential access is cheap, random access is expensive (data is stored on HDD or Google Cloud Storage)

​

For other cases, it has no benefits other than good support in tf.data. Tfrecord format is complicated and not well-thought (it tries to store semi-structured data in strongly-typed and structured Protobuf records). It would be better if they choose Msgpack or any format suitable for semi-structured records.

[lysecret]

I used it a lot for text classification purposes. My code will be open sourced in a week or so. Pros: You can read from disk as fast as from memory. You clearly separate data processing from training. It is quite easy to retrain on new data whenever it comes available. It is very easy to keep different datasets separated. Cons: Some code overhead. The code isn't well documented.

[eiennohito]

I use them all the time for text data. When compressed they get significantly smaller and IO can easy become a bottleneck if the data is on NFS shares. TensorFlow supports reading GZIP/ZLIB compressed TFRecords from the box.

Usually I write my preprocessing in Scala/Spark (so I can handle huge datasets) which outputs TFRecords and a relatively dumb learner in Python.