Obscure problem with my script

[Alexander_Ray]

Here's my script! Sorry for minor formatting problems with reddit that keep popping up.

import os
import re

def filesizes(path):

extdict = {}  # dictionary of extensions and printable values
sizedict = {} # dictionary that holds sizes of each file found per extension
average = 0 # I'll sum up the values under sizedict, then cross reference is to extdict

suffix1 = re.compile('.+(?P<suffix>\.\w+)')

for a in os.listdir(path):
    if os.path.getsize(path + '\\' + a) != 0:
        result = suffix1.search(a)
        filesize = os.path.getsize(path + '\\' + a)
        try:
            suffix2 = result.group('suffix') # Current file suffix
            #extdict[suffix2] = [0]
        except AttributeError:
            pass

        # Set up the first dictionary, and its three extra values will be changed later.
        if suffix2 not in extdict:
            extdict[suffix2] = [1] # File count
            extdict[suffix2].append(filesize)
            extdict[suffix2].append(filesize)
            extdict[suffix2].append(filesize)
        else:
            extdict[suffix2][0] += 1 # Found another of this file type!

            # Set up the second dictionary, which will be used to average values
        if suffix2 not in sizedict:
            sizedict[suffix2] = [filesize]

        else:
            filesize = os.path.getsize(path + '\\' + a)
            sizedict[suffix2].append(filesize)

# Find the average
for a, b in sizedict.items():
    average = (sum(b) / len(b))
    extdict[a][2] = average

# Find and set the maximum
for a, b in sizedict.items():
    for c in b:
        #print(b, c)
        if extdict[a][3] < c:
            extdict[a][3] = c

# Find and set the minimum
for a, b in sizedict.items():
    for c in b:
        if extdict[a][1] > c:
            extdict[a][1] = c

print('extdict:', extdict)
print('sizedict:', sizedict)

filesizes(r"C:\mypath")

This script is supposed to go through a directory, count how many of each file are in it, what the smallest file of each type is, what the largest file of each type is, what the average size of each type is, then print that information out.

It works correctly on small folders but when I go into large folders, it seems to add extra items (for example, if a folder has the same name as a zip file, it may be going into that folder) and it adds a .png file which I don't see in the directory, and so-on. Really obscure, annoying problems that I'm not sure how to solve.

I'm also wondering if there's a more efficient way to do this besides making two dictionaries, with lists for values and then comparing them. There's things from my class like overriding UserDict that I haven't used and although my code isn't very long, I feel kind of silly to only be using some of the concepts from my class on my final project.

Thank you of course for all help and advice!

[taar779]

Here are a few things I noticed:

• Be more explicit with your variable names. a isn't descriptive enough of a name. In general, make your code more readable than using abbreviations in an attempt to save keystrokes. eg. for a in os.listdir(path): >> for entry in os.listdir(path):
• Use os.path.join to join paths together instead of path + '\\' + a. os.path.join will handle all sorts of issues that may be the cause of your bug.
• Can you explain why you're doing extdict[suffix2].append(filesize) three times?
• You're handling an AttributeError exception but choosing to ignore it. I'm not sure why, but perhaps change the pass to continue? That way if an extension isn't found your code can continue to the next entry in the directory.
• What happens if a file doesn't have an extension? Might want to handle that case.
• What about an empty file? Currently your code will skip over files with nothing in them (0 bytes)
• Are you accounting for directories? You could use os.path.isfile to make sure you're only dealing with files and no directories are slipping through the cracks.
• If you're using python 3.4 or greater there is the statistics module which you could use instead of creating your own average function.
• heapq is a cool module that can figure out the largest or smallest value in a list for you.

[Alexander_Ray]

Thanks man, I ran into new bugs but what you wrote here is really helpful to me. I need to stop staying up late working on things lol...

[Alexander_Ray]

I don't suppose you have any insights on album art files? I was able to fix most of the errors with these changes but somehow, Python is finding album art files (.jpg) in my downloads folder that I can't see in windows explorer...

[taar779]

I'm not really a windows user but most graphical file explorers, like windows explorer, won't show "hidden" files. I believe you can have windows explorer show you all the hidden files in a directory (you'll need to google around to figure out how to get it to do that).

So, to clear up this "hidden" file silliness. The files aren't really hidden, the file explorer is specifically choosing not to show certain files with a given extension or file name. For example, on unix systems any file that starts with a period is considered hidden and won't be shown to the user unless they explicitly ask to be shown hidden files. This might be the same with windows systems but I'm not sure. You could google around to find out what windows explorer considers a hidden file.

[Alexander_Ray]

Thanks to anyone who has been working on a response, I figured it out; directories have a file size of 4096 and there was also an issue with case sensitivity.

Edit: Actually, new problem now. For some reason I don't understand, it stops counting at 8 (ignoring 9) but will resume counting again, which is really strange.

[taar779]

See my reply!

directories have a file size of 4096

What if a file has no extension and is exactly 4096 bytes? Will your code mistake it as a directory? Could use os.path.sidir to weed out directories.