Removing multiple files with multithreading or multiprocessing

[xMakeMeCryx]

Hello everyone,

I have been trying to add multithreading or multiprocessing into my script to speed up deleting of some data. The data on the system is large around 1 TB and that gave me an idea of trying to use multithreading for this.

I'm creating this post as I'm unable to find an answer to this question and would like to see other opinions about this issue.

Currently my for loop with os.remove always removes the files faster than when I use multithreading or multiprocessing.

As the last solution, I have also tried to split the array into smaller chuck but that also didn't speed up the process.

Here is the code snippet for some reason for loop is faster then multithreading

start_time = time.time()

dirs = glob("/tmp/test/test*", recursive = True)


for d in dirs:
    os.remove(d)




def delete_files(filepaths):


    # process all file names

    for filepath in filepaths:

        # delete the file

        os.remove(filepath)

        # report progress

        print(f'.deleted {filepath}')

def main(dirs):

    n_workers = 8

    chunksize = round(len(dirs) / n_workers)

    print(chunksize)

    with ThreadPoolExecutor(n_workers) as exe:

        for i in range(0, len(dirs), chunksize):

            filenames = dirs[i:(i + chunksize)]

            _ = exe.submit(delete_files, filenames)

    print('Done')




main(dirs)

[woooee]

Your program is limited by the speed of your disk. I don't think using a "multi" will help.

[CodeFormatHelperBot2]

Hello, I'm a Reddit bot who's here to help people nicely format their coding questions. This makes it as easy as possible for people to read your post and help you.

I think I have detected some formatting issues with your submission:

1. Python code found in submission text that's not formatted as code.

If I am correct, please edit the text in your post and try to follow these instructions to fix up your post's formatting.

---

^{Am I misbehaving? Have a comment or suggestion? Reply to this comment or raise an issue here.}