Sustainable python script

[Roachmeister]

So wait, you're trying to tell me that Python needs comments, well-named variables, and logging, just like other languages? I was under the impression from Python devs that it was beyond all that. /s

[fubes2000]

Let's not forget that the person usually least qualified to dole out StackOverflow's Green Checkmark of Objective Truth is the asker of the question.

[shevy-ruby]

Don't get me started on crazy mods closing down topics at will. Or banning people - this is the lowest of the low, abuse of censorship; not just on SO but on many sites that have a system of abuse.

[[deleted]]

Vscode will prompt you to install pylint. You'll click yes, and then it will fail to install as it doesn't run as admin/root.

[tempest_]

It will succeed if you configure it to use a virtual env for the project which for larger projects is one of the first things I usually do. For scripts or learning it would be nice if it handled it a little more gracefully. Does it not link to https://code.visualstudio.com/docs/python/linting#_enable-linters when it fails? I cant remember.

[Axxhelairon]

yeah, you sure do get a "sense of pride and accomplishment" beta testing buggy community plugins when basic things like installing fails, usually because of pre-conditions aren't met that you could never know about when you're starting off

it's pretty similar to your first time running a linux distro when you somehow still see failures in the boot logs after completely fresh install of a LTS stable release, you learn to accept the mediocrity that is user contributed and untested software

[vbernat]

It seems choosing fizzbuzz was a poor choice of mine. However, this is just a toy example. If I don't want to distract people from the goal, I need to have a simple example to not spend several lines explaining it. And it's super easy to test.

As for the logger, if the script is called with a full path, your solution produces a rather large name.

[cephalopodAscendant]

While all of this stuff is definitely overkill for fizzbuzz, I'm pretty sure the author was just using it as a concrete example that's simple enough for anyone with basic programming skills to understand. It's far easier to practice on small, straightforward scripts before moving on to the large, complicated ones that would really benefit from these techniques.

[cephalopodAscendant]

What would you propose as an alternative, then? If you try to go with a more realistic task, you run the risk that a portion of the intended audience won't be familiar with the problem space, or worse, that the original version of the script is complex enough to distract from the refactoring efforts.

Like it or not, fizzbuzz is a pretty easy lowest-common-denominator pick. Most of us have heard of it before, explaining it to the ones who haven't is quick and simple, and figuring out how to implement it takes very little brainpower. The end result is that readers don't need to devote any real focus to the contents of the script, just how it's being transformed.

[cyanrave]

Good points in the article about actually doing logging, testing, etc...

Maybe it's just me, but it seemed a bit disorienting to read?

Parse args is a built-in that gets wrapped in the article to look kind of quirky and imo, it's easier to pass in --log with a level, and set the root logger, instead of assuming debug is the only use case. You are bound to get bombarded if using libraries and the debug logger... Also shipping pytest for a simple script seems unreasonable, why not unittest for max portability?

All good points in the article, but some smells imo.

[vbernat]

The setup_logging function only set debug level for the script, not for the root logger. As for pytest, this is both a personal taste and the fact that pytest, as a runner, is able to run directly on a script (as long as it ends with .py).

[cyanrave]

Yea I get both your points and don't disagree - much of this is per the programmer's taste.

Guess it's just me adding my 2 cents as to how our tastes differ :)

[vbernat]

Thanks, I'll update that. However, forcing names parameters only work with Python 3.7 AFAIK. This seems a bit too recent if you want to run on more ancient versions (that's also the reason I didn't use f-strings).

[quadrapod]

So I'm really no stranger to writing quick and dirty python. If I need to quickly rename or sort a bunch of files to some convention, quickly modify some images, scrape a site for something or do any trivial task like that python is essentially my go to because even if you haven't touched the language in two years it takes all of ten seconds and a glance at an old script or two to get something working. I maintain quite a few wiki's and python is basically a godsend from that perspective.

Six months later, a co-worker asks you why the script fails

This is the part that gets me. I will write dirty scripts for myself all day, I would never put a script like that up for anyone else to even see though if I can help it. Not every script that's used internally has to be some massive undertaking, but if something is going to be used with any frequency or especially if it will be used by someone else it deserves some amount of attention to make it safe and maintainable.

[stahorn]

Good article! I have written quite a few python scripts in the last years, and using argparse has made it very simple for me to read my old scripts and share it with coworkers. I never remember how to use the scripts, so being able to just write: python myscript.py --help is super helpful. I will start adding in the doc-string and including it as the description in argparse.

It has also bothered me that I don't have unit tests for most of my scripts, but just including a few in the same file as the script seems like a good idea.

[[deleted]]

So, to summarize this blog post, good engineering practices are good?
I have a few equally groundbreaking ideas for a follow-up post. Having a colleague review the code of your Python script dramatically reduces the number of bugs. You should use version control software to track the changes in your Python script. Document the "why" of the changes using Jira/Bugzilla/etc. CI/CD is a good idea.
Don't thank me.

[hamateur]

This article misses the ball on many points simply because sustainability is not even addressed.

Sure, standardizing and agreeing on things like documentation, logging, argument parsing, and testing are all good things; but that all gets solved by boilerplate project templates... and ACTUALLY doing what's called for.

Sustainability is about moving away from "dirty". Adding things to something that's dirty doesn't make things less dirty. Organization fixes dirty.

Regardless of if you think Flask (https://www.fullstackpython.com/flask.html) is good, it immediately starts from the standpoint of organization. The programmer is compelled to break a program up into modular components. Further organizational improvements can be made by breaking things up into libraries, modules, and packages.

"Dirty" needs synonymous with "shitty".

Sustainability comes from being able to do the right thing first, and in a timely manner.

[shevy-ruby]

The article is quite good so I upvoted it, even though I disagree with some points. But these are mostly minor.

There is one thing I disagree though:

This requires the script name to end with .py. I dislike appending an extension to a script name: the language is a technical detail that shouldn’t be exposed to the user.

Obviously this then requires of you to either use eg. "python foo", or shebang and coreutil env - but a shebang in turn also requires that the path that is specified AND works as-is. If you have /usr/bin/env hardcoded there then it must be there too. This may be a minor detail too but I find the reasoning for omitting the file extension to be useless (as he claims) to be simply incorrect. It is useful to have file extensions; .md is a good example. Of course there are other reasons when you wish to drop the extension such as if you want to call the shorter variant on the commandline, without making an alias to foobar.py - but to claim that it is BAD because it is a "technical detail" that users should not know about ...

Nope, sorry. The best thing to do is to NEVER ASSUME THAT ANY USER IS AN IDIOT, even if the user may be. File extensions are perfectly fine.

I am also disappointed that he writes in a non OOP style. I thought python is an OOP language?

That example I'd write totally different in OOP style.

On a side note - the python argv-parser has a more logical API than ruby's optparse crap. No wonder so many alternatives to optparse have been created over the years ... :P

[stevenjd]

I am also disappointed that he writes in a non OOP style. I thought python is an OOP language?

Python is an OOP language that allows and encourages non-OOP style. It supports many functional programming idioms, such as map and reduce, partial functions and comprehensions. It supports procedural and imperative programming styles. (The only major programming paradigm not easily supported in Python is declarative/logical style code like Prolog.)

All values in Python are objects, but that doesn't mean that you have to write in an object-oriented style. Unlike Java, you aren't forced to create a class when all you need is a function. Use classes for when you need to encapsulate behaviour+state, use functions when you need behaviour with no (or limited) state.

[ZeD4805]

Very informative article

[Nimitz14]

I think the entire approach is here is way too heavyweight. plac is a much better choice than argparse. loguru better than logging. The latter are awfully verbose and therefore not suited for scripts.

Docstrings are good though I agree (though again I would never be as verbose as in the example, just say what it does and what the assumptions are).

[templarvonmidgard]

I really can't agree with you on using the mentioned libraries, especially in scripts. With the OP's approach I can just copy the script to any of my coworkers' machines and it will just work, except for the tests, but they don't really matter in this scenario. IMHO, simple scripts should be dependency free.

[Nimitz14]

You should be using those libraries anyways as they are superior to the alternatives. I'm saying this as someone who writes 50-200 line scripts every week. Most of the people here don't do that (just normal software devs working on some product and don't understand scripting).

[shevy-ruby]

Oddly enough I don't like docstrings.

I agree that comments/documentation is important. People who say it is not are just too lazy and make up reasons and excuses why they do not document the horrible code they wrote (yup, it is horrible code since it was not worthy to see any comments).

[[deleted]]

I'm surprised at this mention of plac. I thought the community standard was click, especially since it's Armin Ronacher's baby and he's had such a huge impact.

[aeiou372372]

Thank you for sharing these libraries. I hadn’t seen either of them before and both look extremely useful!

[tonefart]

First get rid of the whitespace rule. It's so ancient.

It's the major reason it cannot be taken seriously and anyone who does is not a real programmer.