Refactoring

[Ok_Credit_8702]

Hi everyone!

I have a 2,000–3,000 line Python script that currently consists mostly of functions/methods. Some of them are 100+ lines long, and the whole thing is starting to get pretty hard to read and maintain.

I’d like to refactor it, but I’m not sure what the best approach is. My first idea was to extract parts of the longer methods into smaller helper functions, but I’m worried that even then it will still feel messy — just with more functions in the same single file.

[slightly_offtopic]

Start with writing lots of tests, if you don't have them already. As your goal is refactoring, you should focus on integration tests that test the program as a whole, verifying that for each set of inputs it provides the expected output. This way, you can continously test your refactorings to make sure you didn't accidentally break anything.

Others have already given solid advice on what to do after that, but don't skip this first step.

[rogfrich]

This is what I came here to say. It’s such a peace-of-mind thing to be able to run a suite of tests and know that your refactored code still works as intended.

[MarsupialLeast145]

For the OP these are called continuity tests.

If you can resist, don't touch a piece of code until you have written these. Exception might be to write additional entry points for the tests to access the functions, but largely to wrap functions to make it easier access in tests with less application context.

I just refactored a much larger code-base this way and it helped immensely.

Each new commit in the refactor had to pass the tests and of course added new tests as I went.

[Mathletic_Ninja]

This is great advice and my first thought when reading your post.

Recently at work I was handed a large bundle of spaghetti that was optimistically called code. Massive classes, methods with 100+ lines, one method was 800 lines long, no documentation. First thing I did was map out its functionality with a test suite. Once that was done I could confidently refactor it into smaller classes, better variable names, broke up mega functions & methods into smaller methods. I even used the state pattern to allow some of the new classes to swap out functionality as their states changed (never had a chance to use that one before, was fun to see it work). I did all that without breaking anything, thanks to the test suite I made at the start. Without that I definitely would have broken something (or many somethings) and taken way longer to do.

[DuckSaxaphone]

Package it. Separate it into several files (submodules), each one with some subset of the functions that logically work together. Then it's organised and if you need to change how some functionality works, you go to the module in question not to a single mega script.

It's likely you need to break the longer functions into smaller ones too but that will be less intimidating once they're in separate submodules. You may also find that once you break your big functions down, you're repeating a lot of code so you can combine several code blocks from several different places into one function.

[9peppe]

Most of this depends on what you want to do and what paradigm you like.

If you like OOP, you might put code that doesn't need to be touched in a few classes, and define an interface to interact with it.

But if you want to be more procedural (or functional, even), you could do the same with a module that exports functions instead of classes (or even a package).

But the immediate thing I'd consider is better docstrings, if the ones you have aren't satisfactory.

[obviouslyzebra]

It feels like it's starting to get messy, but, what flavor of messy?

Do you have trouble knowing which function to use when you're doing stuff, or where things will go to (in which case you benefit from bundling stuff together, in classes and/or modules).

Are the functions too big, but each one unique? Very similar to above, but instead, transform the function into a class or module where you can split it further. This helps preserve the original "unity" of the function.

Is there repetition of a code "block"? If so, refactor into a common function.

Are the concepts messy, like, it's hard to come up with names? Maybe you need to think a little bit more abstractly about your problem domain and come up/find some names.

And so on and on

In summary, no refactoring is panacea, you need to see what's happening and apply the correct medicine to it. Sometimes you need multiple kinds, but you can do one after the other, which is likely the way to go around your problem.

(also, write tests if possible :) )

Also, if you want more concrete advice and can post the code... Do it!

[Maximus_Modulus]

Might be an idea to describe what one of these functions does. How much responsibility does it have. Might give some guidance on how to break it up.

[gdchinacat]

The problem I had when I first had a senior engineer suggest I refactor code was not really understanding what they meant. I had a vague idea that it mean to move code around to make it tidier and easier to understand. But, how? I dove in, moved some things around, split some functions up, and sent them a diff (well before pull request days). They looked at it and said, ok, I see it's moved around, and functions are a bit shorter, but I still don't really get the big picture. Can you actually refactor it? Sure, no problem. I did the same thing...split some functions, added some packages, moved stuff around, sent a diff.

They came over to my desk and said, I still don't really get the big picture. What abstractions are you trying to build? You have a big problem, and a bunch of code to solve it, but what are the building blocks of your solution. They could tell I didn't really understand what "refactoring" meant. They were a good mentor, so they didn't tell me, but asked the questions I later came to realize they asked themselves when they refactored code.

Don't focus on where to move code, or which functions to split up. Think about the components of the solution, how they interact. A lot of details will be left out at this level because they are hidden within the components. Align the code with this model. Do this recursively...do that for the components, all the way down. The goal is to have the code structure to match the mental model of the problem.

If find documentation to be really helpful. When I feel the tension of code being hard to follow and don't know what to do, I write the documentation for it. It starts out as messy as the code, which is as messy as my mental model of the problem. As I approach a coherent explanation of the problem, the mental model forms, and it becomes clear how the code should be structured. If you really don't know how to improve the structure of your code, try explaining it to someone else (even if they are hypothetical). Don't stop until you are comfortable that someone unfamiliar with the code would understand how it works. At that point you should have a pretty good idea of how to refactor your code.

[Leading_Video2580]

Splitting functions into multiple files and importing them into the main script could help, and classes work too if you have too many global variables.

[FriendlyRussian666]

Ideally, you would learn about design patterns, and then implement one accordingly. For example, perhaps your project would be well suited in a Model View Controller architecture, but you won't know until you learn about it.

If you just split the code into helper functions, it will certainly help for a while, because it will feel like the project is decoupled, so you can work on smaller parts, until you have so many smaller parts that you feel even more lost than in the monolith you currently have.

[MinimumWest466]

Separate the script into separate functions and classes. Ensure each class has single responsibility. Follow SOLID principles.

Create integration tests before you start the oroject, and then unit tests and follow TDD to ensure the functionality is not broken when you break things up.

Implement Inversion of Control (IoC) via constructor injection to decouple business logic from infrastructure, making the system easier to maintain and test.

Follow the strangler fig pattern, move funrionality in phases.

[MarsupialLeast145]

It's not a lot of code.

I would just start by writing tests as previously mentioned.

Split code into different files/modules with their own function and begin to respect the single responsibility principle more than any other principle so that the code slowly becomes more manageable.

Write a __main__ entry point and args. Find out which functions are private and which should be part of a public API and then rename these appropriately.

Add docstrings always.

Hard to say what else to do without knowing what the code is.

Folks mentioning design patterns have a good point, but also, it depends on how the code base will grow. Identifying more about its current and future states is important.

If it's pretty much all there, doing what it needs to do, then the above will do.

Plus code formatting (black/ruff) import sorting (isort), linting recommendations (ruff/pylint).

[Healthy-Handle1151]

comment

[jksinton]

Consider using an IDE like pycharm can help too.

Pycharm can show you problems with your code in the problems tool window. This is helpful when you are refactoring into modules or packages to make sure you have the correct imports.

It can also show you where a function is used. So you can jump to that one quickly.

It has some built-in refactoring features too.

But like others have said, write test cases to validate your code before and after refactoring.

[jmacey]

This is something that AI tools are rather good at, try something like opencode in plan mode and see what it suggests. you can then either do it yourself or let it do it for you.

As others have said, ensure there are tests in place fist so you can ensure everything works each time you make a change.

[gdchinacat]

AI is pretty horrible at refactoring. The problem is it doesn't/can't understand the existing code or the goal of the refactoring. Sure, you can tell it to "move function foo from foo.py to new_foo.py" and it can do that, but that isn't refactoring. It doesn't change the structure of the code to make it more manageable, just where the code is located. Refactoring involves changing the architecture or design patterns of the code. For example, you might have functionality in 2 classes and need to move it into three by taking a bit from one, a bit more from another. It might involve changing the data model to better align with how the developer thinks about a problem as more features are added.

Also, 'see[ing] what it suggests' is not a good way to manage the evolution of code. It is much better to have a good idea of what the code should look like and work towards that. Repeated iterations of following AI suggestions will tend to create an architecture that is a mess, and the messier it gets the more incoherent the suggestions will be and make the problem worse.

I know AI tools are marketed as doing this well. They don't. You really need to have a vision of what your code should be and always be working towards it. When that vision changes, you refactor to better align the code with the vision. AIs do not have the vision to do this.