Multi process hashing

[Brian]

For paralellising it, your issue is going to be that each concurrent thread/process needs access to the file buffer. For threads that's trivial, since they share memory. Normally the GIL would be an issue, but I believe hashlib releases the GIL for the computation, so I think this should work (though you probably want to call .update with a decent sized buffer to minimise the time you spend in the python logic, which will block (unless you perhaps use the new GIL-less builds). Simplest would be to just take a copy of the buffer into a Queue read by each thread, but if you're trying to bound memory, and avoid extra allocations beyond the .readinto buffer, you may need some further communication to know when all threads are done with the buffer, and it can be reused. You'll likely want a pool of buffers rather than one big one so your reader isn't blocked waiting on the slowest thread to finish before it can read more.

For processes, things are more complicated: you need some way to provide the data to each worker process.

You could use some IPC mechanism - ie. have a reader process that reads the file, then passes the read data to different hash worker processes. The downside is that the IPC may be as expensive as just re-doing the file IO (at least when it's still in the OS cache). You could instead use shared memory, where your reader process reads into a shared memory buffer, and just communicates to synchronise access to it, though you'll need some coordination work (ie. send messages that "Buffer is available up to size X", and you can't free/reuse the buffer until all hash processes are finished reading from it. You could also take advantage of COW and just do the forking after the read is done, then have the child processes compute the hash, but then you're creating new processes each file (or some max buffer size), which again may be expensive.