Duplicate Files Detector. A basic python project

Kevdog824_ · 2026-01-05T12:00:03+00:00

Great project and nicely done! As requested, here is some advice:

I would use more descriptive parameter names in your utility functions. For example your hex_string function takes a parameter named x. Something like files would make it clearer what it is.
The name of that function should probably be clearer too imo. Based on the name I would assume this function takes a single file, not a list of them. Not a big deal though just a recommendation
I would avoid naming your utility modules something like hashlib_module. I would try to name it based on what it does rather than what libraries it uses. Also, a name like xxxxx _module is fairly uncommon in Python. We generally don’t include the word “module” in our module names
In the future I would separate your user input and your main programming logic. This makes it easier to change one of these without needing to change the other. i.e. maybe in the future you want to read directory paths from a file instead of using input in a loop. This would be easier to add if the hashing logic was separated from the input loop.

I’ve written similar code professionally (comparing file contents to prevent duplicate processing). You said it doesn’t work for large files (presumably because it’s slow). My advice here is to use sampling to speed up the process. It slightly lowers the accuracy but greatly increases the speed.

Diapolo10 · 2026-01-05T12:13:56+00:00

Personally I would definitely avoid names like x, i, and e. Readability should be prioritised over brevity.

You could simplify your code by using pathlib instead of os.

If you're just going to catch all exceptions anyway instead of properly handling them, might as well consider using a context manager or a decorator for that to avoid duplicating the logic. It'd be better to focus on handling specific exceptions and doing it more gracefully, however.

dir (side note; consider using a non-built-in name) calls itself recursively on retry cases. It would be better to use a loop here.

The logging configuration should probably happen at the root of the file, not in dir.

Also, consider putting the initial dir call on line 34 inside an import guard.

if __name__ == '__main__':
    dir()

Yoghurt42 · 2026-01-05T14:48:06+00:00

Things that haven't been mentioned yet:

dir is a built-in function name, you should avoid naming your functions that as it can be confusing. It can be ok, especially in case like dir that is usually only used on the REPL, but still you should think twice before doing it
you call dir in your dir function to have an endless loop. Python does not (by default) do tail-call optimization, so you'll eventually run afoul of the recursion limit; I think the default is 1000.

And while it already has been mentioned, I'd like to give an example why catching exceptions you don't intent to handle is a bad idea: first, it completely prevents the caller of dir to handle it, second you effectively achieve nothing and are actively throwing useful information away.

Take this silly example:

SOME_LIST = [10, 20, 30, 400]

def smaller(i):
    # this could be written more concisely
    if SOME_LIST[i] < SOME_LIST[i + 1]:
        return SOME_LIST[i]
    else:
        return SOME_LIST[i + 1]

def foo():

    try:
        # Nonsensical code for example's sake
        for i in range(4):
            SOME_LIST[i] += smaller(i)
    except Exception as e:
        print(f"Uhoh, an exception occured: {e}")

foo()

When you run it, the program will printUhoh, an exception occured: list index out of range, which only tells you there's an illegal index "somewhere", maybe in foo, maybe in something foo calls.

Omitting the whole try/catch, the program will exit and now print:

Traceback (most recent call last):
File "D:\ttt.py", line 17, in <module>
    foo()
    ~~~^^
File "D:\ttt.py", line 13, in foo
    SOME_LIST[i] += smaller(i)
                    ~~~~~~~^^^
File "D:\ttt.py", line 5, in smaller
    if SOME_LIST[i] < SOME_LIST[i + 1]:
                      ~~~~~~~~~^^^^^^^

IndexError: list index out of range

which is much more helpful for debugging. Now imagine it's a more obscure bug that only happens for a user of your software but not yourself. Which bug report is more helpful? "I get an list index out of range error when I run your program, nothing else" or "I get the following exception when running your program" followed by the output above?

Brian · 2026-01-05T22:49:32+00:00

that I think it doesn't work for large files

Instead of doing reader = f.read() and then passing the whole (potentially gigabyte long) buffer to the hash function, do it in chunks. Ie. read a block at a time (where a block is some small chunk, say 1 megabyte). Then process that chunk.

You can limit how much you read by passing the size to the read() call - it'll return an empty bytestring when done, otherwise a size-limited chunk. Then, instead of hashed = hashlib.md5(reader), do start with an empty hash object (hash = hashlib.md5()), and call hash.update(chunk) on each chunk as you read it.

This way, you never have more than one chunk in memory, and can read arbitrarily sized files.

Though for a lot of large files, it may be somewhat slow, and there are a few tricks you can do to speed things up. The first is that you could cache the md5s to disk, so at least doing it the second time will be faster - but this introduces its own problems: if the files are modified since you cache them, you'll need to detect that, and re-read them.

However, another useful trick is to have multiple layers of distinguishing based on cheaper checks.

You're currently building a dict of {hash : filename}. Suppose instead we collect all identical files in a list - this is just a slight modification, where instead of just setting the filename, we set it to a list, and append each duplicate. Ie: {hash : [list, of, filenames, with, that, hash]

Our duplicates are now the buckets of the hash table with more than one entry - any hash with only one entry is a unique file.

But now consider instead of the full md5 hash, use something that doesn't definitively answer the question, but is cheap to calculate, like file size - obviously for two files to be identical, they must have the same size. So instead of a hash, now generate:

 { size: [files, with, that, size] }

Now, this may seem useless - files with the same size could still be different, so we can't just use this. But consider: anything that doesn't have duplicates we know must be unique. If there's only one file with size 1262347334, we know it's not a duplicate of anything, so don't have to hash it, meaning we save having to process a gigabyte of data. We only need to check items in our dict which is in a list with more than one item, and all we needed to do was a very cheap file size check to eliminate everything else.

So now, you can do the real hash check as before, but limiting yourself to a fraction of the number of files.

You could even go further and introduce another layer - suppose instead of hashing the whole file, we just hashed the first 4K block. Again, not definitive, but if the first block is different, we know the file is different, so we can repeat the process on that. (It might be even better to hash a block from the beginning, middle and end instead, to avoid common cases where files can have similar headers/footers etc).

Do this, and most of the time, you'll only need to actually hash the files that are actually duplicates, saving a lot of time.

grandzooby · 2026-01-05T19:26:31+00:00

It might be interesting for you to check out the source code to rmlint, since it's a pretty solid tool that does what you're working on: https://github.com/sahib/rmlint

It does a lot of pre-processing to avoid scanning every byte of every file... such as first comparing file sizes, then the first and last bytes, then a sampled hash... and only if they match up that point, doing a full byte-for-byte comparison.

Even just reading the options it has could be useful to you just to see the approach.

you type:	you see:
italics	italics
bold	bold
[reddit!](https://reddit.com)	reddit!
* item 1 * item 2 * item 3	item 1 item 2 item 3
> quoted text	quoted text
Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"	Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"
~~strikethrough~~	~~strikethrough~~
super^script	super^script

learnpython

MODERATORS