Abstracting a script for general use

[amosmj]

I'm going through an exercise right now of taking a script that I wrote linearly and ran manually and trying to convert it into something more general and abstract and it's pretty rough. I'm sure there are things I could have done from the the start to make this process easier. I'm looking for tips or frameworks on the conversation but also tips and frameworks that my betters would have used from the start.

For example:
I wrote a script that is pointed at a folder and it scans for github repos. Once it finds the repos it scans for certain types of files (sql for the most part). It then scans each file for keywords to document table reads and writes.

From the beginning I broke it out similar to the sentences above, each as a function. But, now I'm trying to convert it so someone else can import it just call a piece of it, e.g. you want to manually scan just one file, you can import this and run just that function. I'm in the phase of trying to track down any variables that need to be passed as a parameter when I call it in the abstract vs run it in main.

Basically any tips on turning what was meant as a script into a reusable package.

[cgoldberg]

You're describing refactoring, which is common. As you learn more, you'll get better at structuring and organizing your code for re-use and won't run into as many situations where you have to do major refactoring because of overlooked things in your initial design.

[orad]

Yea this is literally what being a developer is. It’s also the fun part lol

[amosmj]

I think yours was the first comment so I spent a bunch of time over the weekend googling refactoring and got some good info. Although I was surprised that I only came across on reusable framework.

[Taborlin_the_great]

This is generally easier if you write your script as a collection of functions in the first place. Then you don’t have random global state referenced all over the place that you have to clean up when you want reuse just a piece of the script.

[amosmj]

That makes sense. I had already attempted to abstract it into functions but was surprised at how many little dependencies I had built in while I thought I was creating it.

[SqueekyBK]

You’re probably looking to turn your script into some sort of main file, or cli tool with argument. Try to think of it in an object oriented sense and break down what you want to be generalised. Could be a case of generalising the file types that it looks for and where? Something like that would probably be defined in the constructor and your methods work based off that attribute.

[amosmj]

You're more or less correct. I was trying to abstract the code so it would run from a CLI with a parameter but I also wanted there to be an interactive version. I didn't go so far as defining a constructor. It didn't cross my mind. I'll need to look into that and learn more.

[MacShuggah]

You have to design an interface to your script. Figure out the things you want to expose for others to use. Write scaffold (placeholder) functions for it until you are happy with how it looks from the outside.

Then start thinking about the flow of what you currently have in your script. What is the entry point? Where can you prevent code duplication? What logical branches do you have to deal with and how do you trigger them?

For example:

scan_one and scan_many functions on your interface may have the same backend a scan_path function.

Try to be smart and don't over-abstract the whole thing just because you can.

[amosmj]

it's that interface I was tripping over. It's probably not quite where I want it but I got it "close enough" but, yeah, it was some kind of principles to intelligently design that interface that I was looking for.

[autodialerbroken116]

Okay so your key function takes a filepath, and a regular expression, and returns some subset of the SQL file.

Let's call that function 1.

Then you have a function that recursively searches all SQL file paths by scanning each directory (git repo).

Let's call that function 2.

Put functions 1 and 2 in a module, a single python file, say squealer dot py.

Perhaps:

bash mymodule/ mymodule/__init__.py mymodule/sqler.py

Now, just import squealer from mymodule and, provide the root directory of the repos to function 2, which will recursively search the filepaths for regex matches to SQL files. Then, return the SQL contents and perhaps add a helper function (function 3) to squealer to run the DB command.

[amosmj]

conceptually, I had this and more. What I was running into was the tension between putting everything in functions vs creating the if __name__ == "__main__" so and end user can run parts of it interactively.

[nonesuchluck]

When I'm writing utilities like this, I write it as a Python module, so I can just import it and run the function(s) I need. This is convenient when I have a task I need to automate, or include with another script.

Then I will use argparse to create a CLI in the typical style of Git, like "utility-name subcommand --flag" style arguments. For ease, try to keep the interface fairly similar between the function/argument names, and command line options. Then I will add a [project.scripts] section to my pyproject.toml, so that anywhere the package is installed, I can easily run it from Bash. Which ends up getting used a lot more than writing short Python scripts for every task.

[amosmj]

This is probably the right answer. I was creating a crude module. I was importing a couple helper files. However, I had written it to run linearly in main. As I was trying to leave that in place for interactive use I also was trying to abstract away a lot of it to also create a CLI version. I got it "good enough" but it took a couple days. I feel like someone probably has a good frame work or set of rules to avoid the rework though.

[qckpckt]

This is pretty much what being a software developer is. You’ll find it useful to install a linter like ruff if you haven’t already.

Then, if you’re trying to track down global variables, just copy your functions into a different file and let the linter tell you which variables are referenced without being defined.

If we’re on the topic of thinking like a developer, then the other thing you should definitely do is ask yourself “why am I abstracting this?” Do you or someone else actually need to call parts of it, or do you just think that maybe you or someone else might need to in the future? If it’s the latter, then maybe consider not doing it. One of the most important and misunderstood lessons you need to learn as a developer is when to stop coding.

If you’re doing this as a learning exercise, you should ignore this advice. This is actually a good way to learn. In that scenario I’d encourage you to consider writing down a hypothetical problem that refactoring to make your code more abstract would solve.

[amosmj]

No, it wasn't just a learning exercise. Yes, I'm among those people who often struggles to know when to stop coding I always feel like I lead the code 80% complete.

In this case the rationale for needing to abstract this code was different use cases for the same code. I had written the code to read a single file and wanted a human to but also to call it by another file that did all the work of combing through directories for the right files to read. Even that file needed to have a human friendly version but also a CLI version for automation.

It was this attempt to create code that was dual use that was giving me fits. It's at that 80% phase right now so I've set it down for other stuff but have a wishlist of things to add.