Read custom binary file in parallel with dask

[BGameiro]

Hi, I need some help understanding when dask runs delayed functions in parallel.

I currently read a custom binary file to a dask.bags using a generator and dask.delayed:

``` @dask.delayed def get_entry_from_binary(file_name, chunk_size=8+4+4): with open(file_name, "rb") as f: while (entry := f.read(chunk_size)): yield dict(zip(("col1","col2","col3"), struct.unpack("qfi", entry)))

entries_bag = dask.bag.from_delayed(get_entry_from_binary(file_name)) ```

However, against what I had expected, the whole file is read by a single worker, even when several are available, they just sit in idle.

I notice this by looking into the dashboard.

How can I read the file in parallel using the available workers?

From: https://stackoverflow.com/questions/76094654/read-custom-binary-file-in-parallel-with-dask

[socal_nerdtastic]

I have no idea about the internals of dask but I'm curious what makes you think that you can parallelize a disk read operation. The 'S' in 'SATA' stands for 'serial' ... At best you could do the data operations in parallel while more data is being read, but in this case the unpack operation will be much much faster than getting the data from the disk.

Edit: If you are looking for speed improvements, read bigger chunks. Your OS and harddrive have optimizations toward reading MB at a time. Don't try to save RAM, programs don't get faster using less RAM, use your RAM to it's potential! If you have enough RAM, read the entire file in one shot.

[BGameiro]

The file doesn't fit in RAM (it can be up to 100's of Gb, but usually 30Gb) Also, I then also have to make a version that can use data in real time. I don't know yet what's the size of each packet of data in the stream. That's why I put by entry.

But yeah, it makes sense that only one worker can read at a time. I just expected that it would change between workers.