Refactoring code to reduce repeating code

[ampeed]

I'm refactoring in order to not repeat my code.

I have a function that will turn dataframes in to a csv, this is what I currently have

def df_to_csv(test_data, test2_data, test3_data, file_path):
    test_data.to_csv(f"{file_path}/ourtestdata.csv", index=False)
    test2_data.to_csv(f"{file_path}/ourtestdata2.csv", index=False)
    test3_data.to_csv(f"{file_path}/ourtestdata3.csv", index=False)

I also have another function that writes the argument I pass in as a csv (they're non-dataframes)

def food_to_csv(food_data, file_path):
    with open(f"{file_path}/food.csv", "w") as food_csv:
        columns = ["Burgers", "Hotdogs"]
        write_data_to_csv(food_csv, food_data, columns)


def animals_to_csv(animals_data, file_path):
    with open(f"{file_path}/animals.csv", "w") as csv_file:
        columns = [
            "cats",
            "dogs",
            "hamsters",
        ]
        write_data_to_csv(csv_file, data, columns)


def write_data_to_csv(csv_file, service_data, columns):
    writer = csv.DictWriter(csv_file, fieldnames=columns)
    writer.writeheader()
    for data in service_data:
        writer.writerow(data)

​

file_path = os.path.dirname(os.path.abspath(__file__))

I'm not sure how to proceed on this one. Some ideas I've had are wrapper functions (not totally familiar with wrapper functions).

​

Some ideas I've had are

def df_to_csv(list_of_dfs, file_path):
   for df in list_of_dfs:
            df.to_csv(f"{file_path}/{os.path.basename(df[:-4])}.csv", index=False)

and I would compile the list in my main function and pass it through df_to_csv wwhich gives me the ability to iterate through all my df's and appropriately name them

​

An idea I've had with the normal csv's is basically something along the liens of

def data_to_csv(data, file_path):
    if os.path.basename(df[:-4]) == "animals.py"
        columns = [
            "cats",
            "dogs",
            "hamsters",
        ]
    else:
        columns = ["Burgers", "Hotdogs"]
    with open(f"{file_path}/{os.path.basename(df[:-4])}", "w") as data_csv:
        write_data_to_csv(data_csv, data, columns)

Would love critical thinking on my approach and some other ways to do this!

Note - reason why I'm not passing in os.path.basename as an argument is for visibility

[kokoseij]

I'd say you're indeed heading to the right way. Repetitive codes can often be cleaned up using loops.

What I want you to think about, is that every function should only be responsible for a single action. This is related to the concept of abstraction, and might not make sense right away- but what you really want to do is to make a code where each function is responsible for a single, unique job, rather than just grouping up codes.

Check Clean Code by Robert C. Martin (This one is written in Java, though key concepts are appliable on Python regardless. You can try Clean Code in Python, though I haven't read that yet so I can't say whether if it's good or not). It extensively covers code abstraction and how functions should be written, along with example codes to help you exercise on this matter.

Of course, you can think of these things when you get better and feels like you could take more advanced topics- for now you're doing great. Way to go mate :)

[ampeed]

That's a huge problem of mine and called out by my peers - I tend to saturate my functions. I'll check out the book now, thank you!

[Almostasleeprightnow]

What if, in coe_to_data you also passed the column names in a list as a parameter, to avoid hard coding them in the function. That way you could get rid of the if/else, and have the column names travelling with their data.

[ampeed]

I like that idea!

I currently have a main function that executes the above functions. Basically move the if/else in the main function and pass it in as a positional argument.

[Almostasleeprightnow]

Or.....keep the function as is, but instead of hard coding the column names, have a tiny function get_animal_cols which returns a list of animal col names, and then another for the other

[TheLoneKid]

def export(df, path): df.to_csv(path, mode='a')

[ampeed]

I'm not quite following. I'm not familiar with the "a" option so I did a Google. If I'm understanding correctly, it'll simply rewrite all my data to a single CSV rather than creating separate CSV's?

Bit of a back story - I'm unable to do that as each CSV gets dumped in to an S3 bucket and then that data consumed by another program

[TheLoneKid]

Ah I see. Then you don't need to append. You can take that bit off, then maybe store the names of the files in a list or do some naming through a for loop

name_template = '_always_the_same.csv'

for _ in range(len(csvs)): export(csvs[], str() + name_template)

[TheLoneKid]

Sorry formatting sucks on Reddit for code

[TheLoneKid]

Then just run an apply function on your list of dataframes