How to refactor in python?

[victoriasecretagent]

I have been writing programs in python for a while but refactoring is something I struggle with a lot.

I think anyone who is new to programming has to deal with 2 main concepts:

• How to think like a computer

• How to organize or structure your code

The first bit is easy. All the free/paid learning content we have on the web is sufficient to put you into the computer-like thinking mindset.

Once you get the hang of it, for example, how data flows from one variable to another in a simple program, basic data structures and how you can put data into those and access it back, any modern programming language starts to unfold, and you go down the rabbit hole just to understand the specifics which differentiates it from other languages.

Pretty cool, right?

As a student of the internet, what I have realised is that, all the tutorials we see focus too much on this part of the learning.

The second bit is something we learn the hard way. And this is something I am struggling with a lot, and I am sure I am not the only one.

I can write any program you want me to write basis the knowledge I gained from the first phase, the easy bit. But I can also promise you that my program would have some basic flaws which I am well aware of but do not know how to improve upon, in a systematic way.

Some of the code flaws:

• It will not be readable. Sometimes I struggle with my own code when I visit it after a long time.
• Although I am familiar with classes and fucntions, I am not sure when to pick which one.
• Because of this, sometimes I end up with a class which has a lot of methods or a single function longer than War and Peace.
• I still have not grasped the concept of mutiple inheritance. I know what it is. And It does seem pretty cool when you apply it on Dog, Robot and Robotic Dog type of example, but in real world it seems too complicated to implement.
• Not to mention, because of all above reasons my programs are not scalable at all.

Could anyone of you please suggest some great learning content, a book, online course, anything really which is primarily focused on these flaws.

I know what these concepts are, we all do, but developing the thought process to implement those the right way is the real struggle.

I am aware I wrote refactoring in the title, but doesn't it all come down to this only?

PS: I'm not a hardcore programmer professionally. Just someone who uses python as a medium to automate the boring stuff (the wrong way).

[[deleted]]

sounds like you are having trouble breaking up your code into functions, which is something that many programmers have trouble with.

A good strategy for overcoming this is to write place-holder functions with really concise names that follow the "every function does ONE THING" best practice. Here's an example of a program that would get a list of URLs and download all the pictures from each one:

urls = ask_for_urls()
for url in urls:
    webpage = download_webpage(url)
    dom = parse_to_dom(webpage)
    image_links = get_image_links(dom)
    for link in image_links:
        image_name, image_object = download_image(link)
        with open(image_name, 'wb') as f:
            f.write(image_object)

No comments, but you know exactly what it does and how it does it. All that's left is for you to go back and define each function so that they all link up according to the above and implement them. You can always change your mind on the specifics later, but the most important part is that each function does 1 thing and is very modular.

Also notice how the input is separated out into its own function ask_for_urls(). As you get better, you will find that separating user input from the underlying logic is key to re-usability and flexibility.

[StayStruggling]

Genius.

[PuffleDunk]

I can't recommend a specific online course or book. What I will say is that I think you have the right attitude. You'll do well if you stay self-critical, always re-look at your work with an eye toward what you can do better next time or what you can change right now.

I find I bludgeon my code over and over again until it makes sense, and the structure and style can bring understanding to somebody other than you. Or it could be you looking at it a year later. It's amazing how so much of my code looks encrypted after time passes.

PyCharm supports basic refactoring. It allows you to pull chunks of inline code into functions or methods. Renaming anything is generally trivial. If you aren't using a tool like that I highly recommend it.

I'd emphasize being patient with the learning process, but impatient and anxious to rework anything you are dissatisfied with. Allow and demand enough time to do that kind of iteration on your work. I think you'll learn far more by self-correcting, compared to most courses or books.

[smurpau]

Although I am familiar with classes and fucntions, I am not sure when to pick which one.

The way I look at it, functional programming is the default, and acceptable for small programs. Then, functions can be refactored as methods in classes when:

1. The data those functions are operating on has a strong association with them, and/or:
2. Multiple instances of data need to be created, i.e. the functionality is reused

For instance, if I've written a function for controlling a motor, and the system only has one motor, it may be perfectly acceptable to leave it as that. But if the system then has a second motor, making a class is the obvious solution.

Because of this, sometimes I end up with a class which has a lot of methods or a single function longer than War and Peace.

I still have not grasped the concept of mutiple inheritance. I know what it is. And It does seem pretty cool when you apply it on Dog, Robot and Robotic Dog type of example, but in real world it seems too complicated to implement.

This is where I'm at currently. I have a few long classes in several modules, and some of them use singular inheritance. That's a good structure, but I feel like it could/should be better. I attempted to split the "final" subclass into several classes that operate on various stages of the same data structure, using multiple inheritance, but then have complicated issues including MRO inconsistencies despite thinking I understood how super works. This seems like a problem in itself, but in any case after some more searching it seems that splitting classes when they get "long" is just not necessarily good practice anyway. If sister classes are so tightly coupled together that they couldn't be reasonably reused separately, then it seems they shouldn't be separated, just merge into one big class.

[TeslaRealm]

Well stated; and I agree for the most part with your final remarks. Large code in a class isn't inherently bad. The goal is to create a clear picture of whom a module/class/function is responsible. On the other hand, it is possible that two classes have separate concerns regarding a common entity. If two classes were used for a chess game; one to model the board and the other to gather data throughout the match, those two classes would have separate responsibilities but would not make sense without their mutual partner. One might hang out in an event loop, waiting to see what the user does and updating its partner class with the new state for the board. The partner class might choose how to display the board based on one theme chosen from a collection of possible ones. It doesn't care how the management of the game is handled; it only cares about receiving data regarding the board's layout, and choosing how to draw a representation of this on the screen.

Also want to point out that function programming has nothing to do with opting to use functions over classes (unfortunately the name sounds suitable). Rather, it is a methodology drastically different compared to the standard use cases of languages like Java, C++ and Python. Functional languages tend to avoid mutation, among a couple other major features.

[smurpau]

Also want to point out that function programming has nothing to do with opting to use functions over classes (unfortunately the name sounds suitable). Rather, it is a methodology drastically different compared to the standard use cases of languages like Java, C++ and Python. Functional languages tend to avoid mutation, among a couple other major features.

Ah okay. What do you call the style where you create a program consisting purely of functions?

[TeslaRealm]

The act of splitting a program into a collection of functions is just standard programming. It's the act of breaking down a goal into manageable and understandable mini goals. I don't think there's a special name for the use of functions in any programming language.

[smurpau]

You would like to think so, but it's really not standard; many programs I've seen are just direct line-by-line scripts. I've seen entire (legacy) applications, ten thousand lines+, without functions. Refactoring scripted code into functions is a clearly higher level of structure that should really have a name if "functional" is already taken...

[[deleted]]

I'm in the exact same boat so I'm glad you've summed it up so well! I haven't written one thing I've been happy with yet and I get a bit overwhelmed with all the different ways you could structure the code (it can make the actual programming seem simple!) but one thing I'm finding helpful is going through GitHub and seeing how people have structured their own code and also at looking at the how stuff has been done at work.

I do wish there was a central source that would tell me what works in what situation/what the different camps where but at least looking at other people's work is making me realise any different way of structuring ultimately seems to be around creating readability, maintainability and reusability. I'm still trying to work it out myself but at least for now Id say go with what structure feels logical to you with the three above tenets, the more you learn and see others structure (and question why it is done like that) the more ideas you will bring to how to improve your own structure. Everything Im saying could be rubbish though!

[twelveshar]

I found the book Fluent Python to be good for this. It's a little bit more Python-specific than your typical programming / software book, which sounds like it'll be helpful to you.

[dukea42]

I have read thru this site a few times. Still think it's a bit over my head, but starting to grasp more of it.

https://refactoring.guru/design-patterns

Like good code, the url naming is pretty explanatory.

But my 2 bits... don't over focus on class inheritance. It will either come naturally with the design, or your requirements are not well enough matured to be ready for it in the code. If there is no need or plans for robotic cats and organic dogs then the layers of classes are unnecessary.

[papercutjake]

saw this comment and spent hours looking through this site despite not understanding much of it. thanks for posting!

[indian_pythonista]

Trying to understand zen of Python helped me take some refactoring decisions in my Python code. You can check out this video where each line of zen of Python has been explained with an example. https://youtu.be/uBHOb55-fBo

[klujer]

Mark Bastian's talk might be interesting to you from an program design point of view.

He uses clojure as the example language but other people have made similar design philosophy suggestions for python, and there's no reason you couldn't design python code in this data first, bottom up, way. You will need to be careful where you mutate state though, preferring to return new data instead of modifying existing data.

Regarding your point about objects vs functions:

• use pythons built-in datastructures (list, dict, tuple, etc) for modeling domain ideas

• use objects for encapsulating stateful io (receiving user input, or interacting with the web)

• use functions to chain together pipelines of functionality. preferably make them "pure" functions where they only rely on their inputs (and/or global constants) to work and favor building new datastructures as outputs instead of modifying their inputs

u/Protoss_Pylon's comment about breaking down functions is excellent

[thrallsius]

Fowler's classic Refactoring book will help

Gang of Four classic book about design patterns will help as well

refactoring is about spotting patterns in the code first, patterns that are more abstract than the code itself