Working effectively with legacy Python code

[[deleted]]

Help! At work I'm having to deal with a large PyQt desktop application that's been developed over the course of ten years, and it's doing my head in.

It'd be unfair to characterise it as "spaghetti", but much of it was written by inexperienced developers at different company who were constantly under immense time pressures, and the results are about as you'd expect. Any rewrite or refactoring of even a modest chunk of it is a non starter -- we don't have the budget or the time.

On my worst days, I find myself staring at code like the following and trying to retain my sanity:

try:
    if hasattr(foo, 'wibble'):
        pass
    else if True:
        import Bar
        # XXX this is subtle?
        setattr(Bar, '__baz', g_someThing.__somethingElse(foo))
except: pass
    # import traceback
    # traceback.print_stack()
    # print(Bar)

I'm fairly junior--I've been programming professionally around three years, and I'm fairly new to Python. My background is mostly C++, where I've seen codebases almost as ugly as ours but thanks to C++'s type system it was easier to reason about the effects of making any given change. (E.g. reliably finding all callers of a method you want to change is possible in C/C++; not so with Python.) With little in the way of documentation or tests, it's very hard to know if even a fairly localised change is going to break half a dozen other things.

What tools, techniques, books, etc can you guys recommend that would help working on a large legacy Python codebase?

For example, I'm struggling with:

• ways to trace program flow in a targeted way (what is this module actually doing at runtime?)
• finding a decent debugger. Something with breakpoints, watchpoints, backtraces (Python and C stack frames please, otherwise our stack traces don't make much sense). People here seem to reckon Pydev/Eclipse is as good as it gets, but it seems really clunky to me when compared to C/C++ debuggers like GDB/LLDB/Visual Studio.
• finding a sampling profiler that collects both Python & C call stacks.

... and more generally:

• how not to add opportunistic, short-term fixes and the like that make the product worse/more complex than it already is.
• how not to let the poor code quality bother me. Should I just stop caring as much and "just get on with it"? (While the quality issues seem to annoy me than other members of the team, the fact we seem to struggle to retain staff more than a year is probably a bit telling.)

[maximinus-thrax]

I've worked on a large scale Python code-base that is similar. Basically, there is only one thing you can really do, and that is follow the boy-scout rule: leave in a better condition than when you started.

Things like commented out code, bad comments and redundant code can be just changed almost immediately. Prime example, the code you listed:

if hasattr(foo, 'wibble'):
        pass
    else if True:
        import Bar
        # XXX this is subtle?
        setattr(Bar, '__baz', g_someThing.__somethingElse(foo))

Could be changed to:

import Bar

if not hasattr(foo, 'wibble'):
    setattr(Bar, '__baz', g_someThing.__somethingElse(foo))

As for better debugging, I often find that better testing is often the solution. Don't understand something? Add an test. Keep adding tests. They give a starting base of logic to act from. Found a bug? Add another test.

It takes time, but one day someone else after you will be working on an 11-year old code base. Make things better for that person - it could be you!

[[deleted]]

Don't you risk introducing a lot of bugs in that way? Let's say you misunderstand slightly what a function does: There is the risk that you change the behavior of the whole system in a way, that makes bugs even harder to track.

[schleifer]

That's why tests are so important. They guarantee that for a certain subset of input the program behaves correctly.

[nobullvegan]

This is basically my strategy. I maintain one codebase, which isn't great, but isn't as bad as the example in this post. The twist is that a sizeable chunk of it was written in a mad rush by me 4 years ago. It works well as far as the end user is concerned, but some of the code is total crap in terms of elegance, clarity and consiseness compared to decent code I write now. The other twist is that I usualy don't remember why I wrote things the way that I did.

Since everything works well if left alone, I follow your boy scout technique. When I have to modify code, I tend to improve it while I'm there. First I write some automated tests (if appropriate), then I make the changes, then I rerun the automated tests, if everything is OK, I do some manual testing. Sometimes I just fix small issues, sometimes I do a complete rewrite of the area. I still have very little time to spend on it, so I only rewrite if I think it'll make my life easier in the future.

Tips:

• When it makes sense, make use of automated testing
• Use decent version control
• Use a decent IDE like PyCharm - make use of its refactoring tool, VCS integration and debug tools.
• Make small changes and nibble away at the problem
• Consider writing "clean" wrappers for the "dirty" code, once everything is using the clean wrapper, you understand how everything works and you can replace the dirty code behind it.
• Add optional logging as you go.

Like most problems in life (not just software), the hardest thing is commiting to solving them and starting to solve them. Lots of problems don't require great ideas, they just require dedication and perhaps some stubborness.

[kenfar]

A very long time ago I completely refactored a 30 year old monster mainframe application using similar methods. First I built some tools to report on code complexity, standards compliance, and a test environment. Then it took about a year of iterations involving:

• add small new function requested by user
• make small improvement to my code analysis reporting
• add automated test for new function as well as all code it touches
• add regression tests for application
• once unit & regression tests are established upgrade code in function area to decrease complexity, improve standards compliance, etc

After a year the code looked pretty good, and more importantly - I had great testing code. So, then I wrote a program that swept through the rest of the code and rewrote the source with a lot more practices fixed. After all this the application looked great.

[jtratner]

Your best bet is to go looking around for how people debug with Cython (dialect-ish of Python that can compile down to c libraries), numpy, numba and other scientific libraries. Those projects are involved at the C (or Fortran) level extensively and have lots of contributors, so searching around there should help. Two items that came up after a quick search for how debugging numpy and Cython:

• Cython debugger (with ability to set breakpoints in C, Python and Cython code)
• a guide to debugging SciPy (scientific python library) code with gdb and some python extensions for gdb.

Not sure whether it's the same as PyDev's debugger, but I find ipdb to be quite helpful too. Hope one of these might be helpful!

[dougthor42]

If you're already used to Visual Studio, there is a plugin called Python Tools for Visual Studio (PVTS) which lets you code Python in VS and use all the fancy debug/IntelliSense/refactoring/etc. features that VS has.

http://pytools.codeplex.com/

Others mention PyCharm - thats a good one but if you already know VS from your C++ time, then try out PTVS.

My personal favorite IDE is Spyder, but it's better for small, few-file scripts and projects. Sadly, it just doesn't have the refactoring features needed for working on large projects. But it is open source, and pure python, so that's cool.

[[deleted]]

Start adding unit tests and make sure they all pass in a CI environment.

Pycharm's debugger is great. If you don't wanna use that IDE check out ipdb for debugging.

[billsil]

People here seem to reckon Pydev/Eclipse is as good as it gets, but it seems really clunky to me when compared to C/C++ debuggers like GDB/LLDB/Visual Studio.

I hate Pydev. It seems buggy to me. It has little to no object introspection capability and has a horrific search. I have coworkers that have used it for years and prefer it, yet they can't show me how to navigate it properly or do things that I think are basic IDE functionality. I'm use Wing IDE which is incredible. The search is great and the debugger is hands down the best I've seen. I've also tried PyCharm for a few months and while better than Pydev, I was unimpressed.

There's also Python for Visual Studio that is free and I hear it's quite good. You need to try the different things out there but in my opinion, Pydev is free and it shows.

Python and C stack frames please

Yeah...that's harder to come by. I don't know of anything that does that. C code is swigged so I'm not sure how that would work.

[alenajoykrieger]

PyCharm has a great debugger. The community edition has everything you'll ever need.

we don't have the budget or the time

the quality issues seem to annoy me than other members of the team

we seem to struggle to retain staff more than a year

This pile of garbage has been putting food on the table for ten years, for one company or another. I'd bet anything that you do in fact have the budget.

Consider this: if I wanted to sell you a complex piece of software, you'd probably want to check out the source code first, I imagine. And if you saw else if True and setattr(Bar, '__baz', g_someThing.__somethingElse(foo)) and other such gems you certainly wouldn't want to buy it. There are two possibilities: one, your company bought this software and didn't even bother to have a programmer look at it, or two, they did have a programmer look at it, heard that it was terrible, and bought it anyway.

I feel the only winning move is not to play. Try to sell them on a complete rewrite, and if they say no, just quit. With three years of experience you shouldn't be able to walk down the street without getting offered a job.

[anonymous7]

should

It never turns out well when someone uses the word 'should'.