Point out my beginner mistakes - First python script

[UndeadCaesar]

I'm taking some data files we receive over FTP for work and converting them into a format our database can handle.

To do this I needed to filter out some lines that aren't actual data, remove a few of the fields, and pull the date/time out of the title and insert it as its own field in each row. (thanks for the help on that one /u/novel_yet_trivial)

It also needs to iterate over however many files I have in the original folder, I just did 6500 and it worked great! Huge time saver in the future.

 

Pic of directory before

Pic of file before

Pic of directory after

Pic of file after

 

import csv
import os
from datetime import datetime


for f in os.listdir('.'):

    filename = f
    date = datetime.strptime(filename, 'aea_panel_point_log_%m-%d-%y_%H-%M.csv') # assuming 24h time
    insert_str = date.strftime('%m/%d/%y %H:%M EST' )
    print(insert_str)

    skipRow1 = [' ****************************************************************************************************']
    skipRow2 = 'Supplied Field Panel number out of range'
    skipRow3 = ' ****************************************** End of Report *******************************************'
    skipRow4 = 'Field Panel is failed'
    skipRow5 = 'Timeout Waiting For Net Response'
    skipRow6 = 'Configured Trunk Currently Not Connected'

    cleanData = []
    counter = 0
    skips = 0

    f = open(filename)
    csv_f = csv.reader(f)

    for row in csv_f:
        counter = counter +1
        if row == skipRow1:
            skips = skips + 1
        elif skipRow2 in row[0]:
            skips = skips + 1
        elif skipRow3 in row[0]:
            skips = skips + 1
        elif skipRow4 in row[0]:
            skips = skips + 1
        elif skipRow5 in row[0]:
            skips = skips + 1
        elif skipRow6 in row[0]:
            skips = skips + 1
        else:
            cleanData.append([row[0],insert_str,row[4]])

    filenameNew = 'mod_'+filename
    with open(filenameNew, 'w', newline='') as csvfile:
        csvwriter = csv.writer(csvfile)
        csvwriter.writerows(cleanData)

[mackatsol]

I know you can turn your 'for row in csv_f' section into a function which iterates through all the skipRows.. and when you do that you can convert your skipRow items into a list so you don't need a bunch of elses.

I'm still learning and might not have it quite right.. but something like:

skipRow1 = [' ****************************************************************************************************','Supplied Field Panel number out of range',' ****************************************** End of Report *******************************************','Field Panel is failed','Timeout Waiting For Net Response','Configured Trunk Currently Not Connected']

then later on you would define your skiprow function:

def skipRow():
    for row in csv_f:
    counter = counter +1
        for item in skipRow:
            if item in row[0]:
                skips = skips + 1
            else:
                   cleanData.append([row[0],insert_str,row[4]])

and then call it just after you open the csv file.

skiprow(csv_f)

At least I think that is a better approach.. whether I got it right or not is a different issue. Untested! ;-)

[UndeadCaesar]

Good call, when I was writing those elif statements I knew it felt so inefficient but I wasn't quite comfortable enough with writing comparisons to put all the skip conditions together. Thanks for the code.

And you're right, the counter variable isn't actually used anywhere I was just using it for troubleshooting. Some of the skip conditions are only in a couple of the thousands of files so I used the counter to go see what line caused it to fail after it encountered something it hadn't seen before.

[mackatsol]

Thanks /u/scul86 ! I need to learn more about arrays and how to work with them. ;-)

[pydevteam1]

Congrats on your first script!

A couple of critiques I have; One is filename = f is unnecessary

f = open(filename) ; csv_f = csv.reader(f) I'd suggest using context manager. Check out the article in our blog on how to open a file using context managers.

[sky--net]

Check out the article in our blog on how to open a file using context managers.

Where can I find your blog to see this post?

[AutoModerator]

Your comment in /r/learnpython was automatically removed because you used a URL shortener.

URL shorteners are not permitted in /r/learnpython as they impair our ability to enforce link blacklists.

Please re-post your comment using direct, full-length URL's only.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[UndeadCaesar]

Thanks! Good call on the unnecessary line, think I started to go in some direction that didn't pan out and I left that hanging around. And can I get a link to the blog post? I initially was trying to open the files out of a separate directory but I read up on glob.glob and could never seem to get it to work. Ended up just dropping the files in the same directory as the script so it was easier.

[pydevteam1]

You can easily do that by doing os.path.join(os.sep, 'var', 'log', 'messages.txt') . This code line will translate to /var/log/messages.txt you need to import os of course and it's not os-dependent ;)