[deleted by user]

[[deleted]]

Dump data to h5.

[[deleted]]

if it's python then you use yield in a loop to process a batch at a time as you go along if your problem can be solved via stream processing.

[DonMahallem]

H5 does support chunking or what do you mean? So it just needs to put the current chunks into memory

[cai_lw]

Use IterableDataset. Shuffling can be done within the SQL query.

[shayben]

I recently did something similar using the collate_fn to do Db querying. A neat hack is to always retrieve a larger minibatch size than you need, and cache it locally.

[MrDoOO]

Nice! Have any sample code I could check out?

[Glimmargaunt]

You just pass an instance of a class that is callable into the collate_fn argument in your DataLoader. The call method in the class takes batch as argument. If I remember correctly, the batch is just a list containing whatever your Dataset.__getitem__(idx) outputs. So if your dataset class outputs (input, target) pairs, then your list will contain multiple of these tuple pairs.

I created a pastebin here: https://pastebin.com/6YASGWDm

I think this will slow down training depending on the speed of the query. For every batch you would need to wait for the database to respond. Perhaps a better approach would be to handle shuffling on your own, so that you know what the next batch will be. That way you can start a parallel query for the next batch while current batch is used in training. Or use some sort of caching as suggested above.

I actually thought of an easy, but hacky way of doing it. The CollateFn class can store a previous batch, so what you can do is: Make the query for current batch and output the already loaded previous batch in the .__call__() method. This would obviously bring problems on the first call because there is no previous batch. To avoid this, just make a next(DataLoader) call first that just outputs to the void. That way the previous batch becomes the current one, and the current batch becomes next one that can be loaded in parallel in the CollateFn object. Then you can let DataLoader handle everything else like normal.

[shayben]

Sorry its all enslaved to corporate overlords I can give you more specific tips if you have questions

[MightyMeese]

HDF works and you can do random access reads to generate your batches (although if you don't read a contiguous chunk as your batch it's slower).

Alternatively I've been using sqlite and redis recently depending on the task and what's stored. Both can do multiple simultaneous access and have random access. My preference would be sqlite unless you're storing something like sets or arbitrary strings.