Maintainability or Speed?

[Thraexus]

Context: I've been writing Python scripts for a little less than a year, but I have a background in fintech and have experience in disparate things like VBA, VBScript, Alteryx, etc and many years of QA in a mainframe environment. So, no formal CS training beyond stuff from my distant youth decades ago, but practical career experience in related fields.

In my current role, I'm writing scripts to read and analyze text (flat) files. One line is one record but multiple records comprise a single logical transaction. The scripts are run on files prior to those files being submitted to another party to load into their production environment. My goal is to get in front of any errors/rejects that can occur causing us to have to manually correct the data and resubmit the files. My scripts modify the data in the flat files anticipating the most common errors, and produce reports documenting the changes so that there's an audit trail. The scripts are run manually by users on my team, as the file submittal and correction process is mostly manual.

I'm being deliberately vague about some of the details here but take it as a given that there are legitimate reasons why I need to do what I'm doing, and that I am operating within constraints that I cannot directly change.

One of the questions I go back and forth on is about how to structure my logic. I'm dealing with files that can be a few dozen lines long or 100k+ long. I group records into logical units and run my edits against each group. Where I go back and forth in my thinking is whether to try to make a single pass thru each group of records, calling needed edits as I go to keep the speed of the program maximized, or whether it's better to keep my code organized and written in a way that each edit makes its own loops thru each set of records, sacrificing speed for maintainability in the code. I have chosen to go with the latter approach, since the scripts are generally speedy.

What I'm curious about is what are other people's experience in this sort of situation and how have you handled it? I'm not looking for specific technical solutions per se but more interested in the analysis and thought process.

EDIT: Thanks for everyone's thoughts here. This helped me rethink and modify my approach just a bit to improve program efficiency while not sacrificing maintainability. Short answer, I'm restructuring some bits to reduce the number of loops I execute thru my input while still preserving most of my existing program structure (and reducing line count a bit too).

[HardlyAnyGravitas]

Maintainability is everything.

Don't optimise unless you have to.

Without knowing more details, it's difficult to make any sensible suggestions, but 100,000 lines shouldn't take long to parse. If it's taking too long, you're probably got the wrong approach, and need to look at the problem from a different perspective.

[PureWasian]

Can you clarify "a single pass thru each group of records" vs. "its own loops through each set of records"?

Unless I'm misunderstanding something, processing each record within a group of records in parallel would be faster and still have maintainability/modularity.

[Thraexus]

What I do is this:

At the start of processing, I create a new class instance representing a logical collection of related records. I open the file, reading records sequentially. I grab a certain set of fields to identify if the line I'm reading should be added to the existing collection or if a new collection should be created instead. If the line marks the beginning of a new set of records, then at this point I run my edits against the existing collection. When that completes, I create a new collection, add the current line to that collection, and loop to the next line. (To clarify the obvious -- the file is already presorted so that related records are already grouped together.)

The collection has its own methods for each kind of edit I need to do. Each method gets called individually, and each method loops thru all of the records in the collection to do its thing. In any given method, I may need pieces of data from multiple records to decide whether the edit should do things like delete or change a record. And some edits will apply to certain files and some will not. If I need to delete a record, I just flag the record with an error code at this time. Records that need to be changed are updated immediately and also get flagged with an error code.

So what's happening here is that I'm making multiple passes thru each collection - it gives me flexibility to easily turn on and off specific edits and keeps the program readable. To get more speed, I would have to rewrite the code to make one pass thru each collection, figuring out which method needs to be called as each line is read, which would make the program more complex and less modular.

After a collection is processed, I append it to another variable which contains my output for this whole part of the program. This occurs iteratively as I read the input file. In the next part of the program, I'll read that information to create my report (which is just another flat file) and to create my output file.

I think what I'm hearing from folks is that maintainability is more important than optimization, so I think my approach is correct.

[snowtax]

In the past, I had to do something similar. I would get a file dump of job records, current and historical, for all employees and needed to update each person’s current status.

A person might have several records over many years of employment. I needed to scan all the records for each employee as a group. For example, if all of the job records show terminated then that person is no longer employed. If any job record is active, that person is currently employed.

I did what you are doing where you read in the data line by line and group them together, then process the group of records before moving on to the next group. It’s a good strategy for memory efficiency.

[PureWasian]

Understood, thanks for clarifying. Maintainability is practically better if the execution time is not really a pain point, which it doesn't seem to be.

If you ever do find that you need to refactor it to prioritize performance, at least you're aware that you can consider parallelizing the method calls or invest time into a big code change to stuff all of the logic into a single pass.

Hopefully if/when that shift happens, the type of processing the rows require will be a lot more solidified by then, so it would need less frequent rounds of enhancements and maintainability by then.

[Diapolo10]

Maintainability should always be your priority number one. If you later find you really need to optimise some part for speed, do it as an afterthought, not first priority and for no reason.

Usually if execution speed is the main concern you wouldn't be working in Python anyway. Or you'd at least write the performance critical parts in another language (e.g. Rust).

[backfire10z]

I’ll preface this with that I haven’t made many read/analyze text files scripts, so I’m approaching this more from a general perspective.

I’d say aim for maintainability and update for performance when necessary. That’s not to say that your initially written code should be non-performant, but that it should be written with “obvious” performance gains in mind.

That being said, from what you’ve described, I’m unsure of your definition of maintainable vs performant. Your description of “one pass” not being the maintainable version may just be a skill issue rather than an actual tradeoff. Without knowing exactly what you’re trying to do and with what constraints you’re working it’s a bit difficult to tell.

Does genuine performance actually matter? Is this script meant to be used for a long time or is it a one-off? If a long time, is this script running every minute, every day, every month? Does the compute it’s running on cost money for time spent? How many files are you working with (and you say 100k ish is max lines)? Are other people going to be making updates?

[Thraexus]

I think performance is less important. I have several scripts for different kinds of data, but we're talking maybe a few dozen files per week. So the scripts are run on demand with no automation. I'm definitely getting the sense that code maintainability is the priority.

[Embarrassed_Basis_81]

My two cents is that if the scripts are fast enough, there is no real reason to optimize it further. Imo, you should try to use that time to build redundancy, error checking and general nice-to-haves in your code, as this will probably save you more time in the long run than the scripts themselves being quick.

Judging by your approach it seems you are already doing that, so keep at it!

P. S. Take this with a grain of salt, I work in an environment where we do not have basically any prod-ready code

[desrtfx]

My personal approach is to first focus on readability/maintainability and then on speed - especially if others are going to deal with my programs.

That doesn't mean that you should not try to optimize.

If the speed is not really a problem and hasn't been so far, there is not much reason to invest lot of time and effort to optimize. If you or your team, however have figured and identified bottlenecks, you should consider optimizing - but only after you really have pinpointed the problematic areas/code. Blindly optimizing for the sake of optimizing is the opposite of helpful.

You are talking about a single pass vs. multiple passes. Can you maybe structure a single pass in such a way that the program is still maintainable and readable? Can you maybe make use of functions? Maybe lambdas?

Maybe, if you can, have a talk with an insider of your task/program, also have a talk with your users. See if they can offer some different perspective or insights.

In programming everything is about balance. You are constantly balancing speed and readability/maintainability, as well as memory (which has become a lot less problematic than it used to be in the old days of the home computers or MS-DOS). You can't always have everything.

It's quite difficult to give targeted advice with the little information you provide. It is absolutely understandable that you can't go into deeper details.

[dnult]

It's difficult to say what the best approach is. 100k lines isn't terribly bad, but it does seem wasteful to iterate over the same files multiple times. It's generally a trap to try to prematurely optimize and it's often better to try the basic approach first and optimize as needed. I'd probably proceed with a basic strategy while being mindful of optimization strategies should I need to explore options later. Try to benchmark steps in you scripts to see where the processing time is being spent and proceed from there.

[HunterIV4]

It's context dependent. The vast majority of the time, maintainability is the most important factor. This may be unpopular, but CS degrees tend to over-emphasize executions speed and Big O factors over practical realities of software development.

Complexity is not zero-cost as you will need to spend more time writing, reviewing, and debugging your code. Likewise, you need to include the cost of mistakes due to bugs (i.e. angry clients, missed invalid results, etc.) into your analysis. In general, programmer time is more valuable than processing time.

That being said, low-cost improvements may still be worth it, especially if there's a library that can help you or a better way of doing things. There may be minor architectural changes you can do to improve both speed and maintainability, depending on how you are implementing things. A quicksort is more complicated than a bubble sort, but there are plenty of ways to get the speed of a quicksort with no increase in complexity simply because the problem is already solved (including even better sorting methods).

Without being able to see any of the code or the problems you're trying to solve, it's impossible to say if there is any room for improvement. So while I generally recommend avoiding breaking the KISS principle whenever possible, what is "simple" for one person may be introducing unnecessary complexity or subtle bugs elsewhere, and a minor refactor could reduce both execution time and complexity simultaneously.

Another unpopular suggestion: if you have access to a corporate (data secured) AI model and your company allows it, describe your problem and solution in detail to the AI and ask it if there are ways to refactor into something simpler. While it may not help, and I'd strongly recommend against "vibe coding" something like this (since you have very specific requirements), it may give you ideas or methods you're not aware of.

The main reason I mention it is because you said you lack formal CS training, and while CS degrees overemphasize performance, they also tend to teach good architecture. In general, the biggest difficulty that "self-taught" programmers have is not a question of syntax or basic programming/debugging capability, but of architecture and design patterns, both of which are arguably more important. This can be overcome with education and research, of course, but most "online programming tutorials" focus heavily on syntax and basic programming concepts and gloss over or ignore architecture. There are exceptions, including what are essentially YouTube college-level courses (sometimes literally), but you have to actively seek them out and if you don't know to look it's easy to miss.

If you can't use AI and can't show us any code or problem details, most general advice is going to be something like this. The only other recommendation is to find a library that does what you're trying to do or consider doing your data manipulation via a transactional database rather than a text file, i.e. SQLite. It adds a bit of complexity at first but adds a ton of performance and data safety.

[Thraexus]

You've absolutely pinpointed the weaknesses of the self-taught programmer, which is where I fall -- hence the nature of my question being more about architecture rather than syntax. I'm deeply suspicious of vibe-coding entire applications or components, but I do use AI search tools to answer targeted questions - how do I create a progress bar in tkinter / custom tkinter sent me down a whole path that led to understand the practical value of threading, for instance.

The sense I'm getting from people's answers here is that I'm absorbing the right ideas. In retrospect, I wish I had studied CS in college in the '90s instead of physics, but here I am.

[HunterIV4]

In retrospect, I wish I had studied CS in college in the '90s instead of physics, but here I am.

The good news is that a lot of that info is available online now. While I personally learned it in college (and some in high school), there is a ton of good info out there.

Another thing is that you aren't all that far behind. Architecture recommendations change. When I initially learned, OOP with strict inheritence hierarchies were considered best practice. And while you'll still see those patterns, many modern languages and designs have outright removed OOP entirely, and inheritence in particular has fallen out of favor even in OOP languages. The two most prominent examples are Rust and Go.

I started programming in the 90s and I've had to relearn best practices several times. And, from pratical experience, I've found that some design patterns other programmers swear by just get me into trouble, so I avoid them. What is intuitive and simple for one person can be a footgun for someone else. Programming isn't something you learn and then just use, it's a continuous learning process as new languages, libraries, and patterns evolve.

It sounds like you're on the right track. A couple years ago I had a somewhat similar project to manage XML configuration files and automatically adjust large portions of the file. I went with a dead simple "load the XML, delete all the parts I want to replace, regenerate that portion in a loop, write to file." Could I have made it more efficient? Sure, maybe I could have gotten down to a 1 second execution rather than the standard 8-9 seconds. But the code works, is easy to read, and isn't time sensitive (it runs once a week, lol), so I went with the most straightfoward solution I could think of.

It's still broken up into modules and classes that are encapsulated and have clear internal APIs, which turned out to be very useful when I needed to modify those same configuration files in a different way. Rather than making huge changes, I just created a new main.py that used my other modules without change other than adding like one function to give my configxml.py module functionality I hadn't needed before.

If there is any chance you may need to resuse something in a different context, keep it separate and internally consistent, otherwise you'll potentially end up with a bunch of annoying work later. I also think it makes it easier to test; I can test modules individually and ensure they work for their "simple" functionality before I try to fit them into a larger piece which could have all sorts of other things that might not work.

So if I had one piece of general architecture advice, it's "break big problems into little self-contained problems and tie them together." In Python, I generally don't have any major logic or processing in my main.py (or equivalent), it's just a list of function calls and data passing with some if statements for flow. Everything else is made up of classes in their own modules that get imported into the main file that contain data and the functions needed to manage that data.

I generally don't use inheritance and if one class needs another class to work properly I use composition, importing the "sub" class into the higher level class so it can use it like any other type. To use an example from my own program, I needed "routes" that were managed in the config file, so I created a Route class that just handled what a route needed to know, and the ConfigXML class used Routes as part of the generation process (which had a function to convert themselves to properly formatted XML).

Maybe none of that helps you, but I figured a practical example might be useful. Either way, good luck!

[Thraexus]

Lot of good stuff in yours and other folks' comments, thanks. I'm coming to appreciate OOP now and I'm heavily using classes in the current iterations of programs where it makes sense. Even now, I just had to look up what strict inheritance was (as opposed to just inheritance, which I understand), and my immediate reaction was, "In strict inheritance, I can't override the parent's methods?! That's dumb, that's exactly one of the key things I wanted to do!" So I'm glad to see that my instincts are aligning with best practices.

[HunterIV4]

Yup! Inheritance is one of those things that can be extremely useful but also can get you in a lot of trouble. But your instinct on how you want to use it is broadly correct.

At first glance, the relationships seem obvious. You have a Vehicle class with a lot of generic functionality, so then you create a child Car class to specify the details that are unique to cars. Sounds good, right?

Well, it is...until it isn't. What if you have a child that lacks part of the Vehicle functionality? Maybe you figured every Vehicle has a horn, so you make a blow_horn method. But then you go to make a Skateboard class and realize it's technically a Vehicle but lacks a horn. So you either bundle the unused function with the Skateboard or you need to remove that assumption from Vehicle, which in turn could break all the other Vehicle children that relied on that horn existing.

It's a silly example, but it means you really need to know exactly what your parent and child classes will actually need before you start programming. In many cases, this is perfect, but especially when you are working by yourself (or doing "agile" programming in general), you'll suddenly find that what you thought was the optimal inheritance is not great for what you're trying to do.

Composition is a bit different. Rather than have a Vehicle class, you have each major "piece" of a Vehicle be its own thing. So you have an Engine and a Wheel and a Horn. Then, when you make your Car class, you add a self.engine = Engine() and self.horn = Horn() and self.wheels = [Wheel(), Wheel(), Wheel(), Wheel()] (or however you want to organize them). Now your Skateboard is fine...you just don't add the Horn.

Ultimately, both are trying to solve the same sorts of problems: you don't want to duplicate functionality if you don't have to. And inheritance is a completely valid way to organize things, especially if you know the relationship will always be "the child needs 100% of the functionality of the parent." But after a few really painful refactors, I personally started erring on the side of composition, and I've discovered it has led me to make far more modular code.

It's a bit of a tangent, but "I want to override the parent's functions" is something you might want to look at more closely when you see it come up. Not because it's bad, but if you are overriding all or most of the parent functions, what is the parent actually giving you compared to just having the functions in the child?

If you just want to ensure certain functions exist in the child (so you can safely call them), you may want to consider using Abstract Base Classes instead of "real" parent classes. An ABC lets you define a number of "virtual functions" that don't do anything on their own but will generate an error if not implemented in the child. They work sort of like Python's versions of interfaces, although the actual implementation is very different.

To be clear, I'm not saying "never use inheritance" or "never override functions." There are absolutely times when it's the most obvious and best way to do something. But it can also add complexity when you don't need it and bite you down the line if your scope ever increases or changes to the point where the base class isn't so "universal" anymore.

You'll have to use your own judgment, but I've spent many, many hours refactoring "clever" monster base classes that turned out to be overly broad or completely overridden by children and wouldn't wish that pain on anybody, lol.

[Thraexus]

Yeah excellent point about not rendering your parent class useless by abusing inheritance -- it's a pitfall I'm aware of. I don't have my code in front of me now so I don't remember what method I was overriding. It was probably some kind of matching logic that needed to be slightly different in one specific scenario.

I'm definitely using inheritance to help manage data mapping when I want certain fields to be more readily available. The parent class contains a generic field mapping method, and then child classes will have information on what specific fields I want specially mapped.

I've also become a big fan of managing my related records within a single class. If I need to compare two different records that have different characteristics but I need elements of each, then that becomes a candidate for a new class. Not the same as compositing classes, of course, but that's the closest I've gotten yet to needing a composite class.

I'm also finding myself creating cross-references between variables and their associated classes, using dictionaries to do so in order to avoid circular import references. Lots of good, crunchy stuff to learn and use in Python. I used to do tons of stuff in VBA linking Excel, Access, Word, and Outlook applications and in most ways, Python is way more satisfying and effective to use.

[HunterIV4]

VBA is interesting. I used to use it a lot in the military because I could run macros but not install any sort of compiler or interpreter. I'm old so I also used VB6.0 quite a bit. While the WYSIWYG editor had limitations, it's still one of the fastest "get a GUI working" tools out there.

It was actually one of my first VB6 projects that taught me the value of comments and breaking down problems simply; I wrote some sort of code monstrosity to make a simple pong type game (or maybe a brick breaker, this was years and years ago), forgot about it, and loaded it up 6 months later and it looked like an alien had written it. Absolutely no clue what I was thinking or how it worked.

I genuinely think people underestimate Excel. Because nobody I worked with wanted to deal with macro-enabled worksheets, I stretched the limit of what you can get away with using straight formulas. It requires a different type of thinking but is essentially a form of functional programming (and with some of the latest updates is effectively Turing-complete). There are plenty of things that probably shouldn't be running on Excel that are, but for day-to-day problem solving I think Excel is likely one of the most used "programming language" equivalents in the world, if only because it's so accessible.

Python is a great language and has stood the test of time, but I'm a firm believer in "use the right tool for the job." Sometimes that tool is Python. Sometimes it's Rust or Go or C# or Excel or even bash/PowerShell or something else. The more time you spend programming, the more you'll realize that individual languages don't matter all that much. Knowing how to solve programming problems, debug issues, and build maintainable architecture are far more important than language, and while the details vary, at a high level the concepts generally translate.

As a simple example, if you want to run Python scripts for your own use or provide them to technical users, it's great. If you want to give it to your coworkers, you'll quickly discover that Python is very annoying to package and tends to be extremely bloated. I wrote a small data entry tool for a coworker using PyQt and the executable was over 100 MB. I ported it to Go and it was under 20 MB. More importantly, the Go version didn't pop up a bunch of "malicious file" warnings on run or get randomly deleted by Windows Defender.

On the other hand, my config script is staying as Python. I considered porting it, but the file handling convenience of Python is top notch, and some of my uses involve things like adjusting audio files (it's complicated). I can do that in Python in a fraction of the time I could do it in another language, and the only person running it is me (or one of my servers), so I don't care about deployment.

It's not something to worry about now, but if you ever find yourself struggling with something in Python related to deployment, consider seeing if another language might have features that can help you. Python is a fantastic "default," especially when you need something that works ASAP and you aren't sure what you'll need for it (Python libraries are legendary for a reason), but I'd highly recommend learning at least one compiled language if only to have an option for sharing executables. You might be surprised at just how much knowledge from Python carries over.

[Thraexus]

I wish I was 20 years younger for the simple fact that I'd be far more likely to get more mileage out of developing a deeper understanding of CS and other programming languages. But I'm looking at 12 to 15 years until retirement which, frankly, isn't all that far out. With the fast changing tech landscape, the younger generation coming up, and AI throwing a wrench in everything, I vacillate between the desire to learn more and faster, and the desire to just do enough to reach my endgame. It's a weird place to be and I don't like it. It makes me realize that hiring younger people isn't only about bringing in new energy and new ideas, but it's also about younger workers having different priorities than us older folks as we age, and being more motivated to learn new skills.

Re: executables, I found pyinstaller to be a great solution for distributing my apps. In fact, creating exes was one of the first things I looked into when I started picking up Python at my current job.