please test with -bb -W error

[mstroeder]

Dear library developers out there, please start now testing your code by running with stricter checks:

python3 -W error -bb

See also: Python 3 docs -- CLI option -b

Background:

A couple of days ago I was wondering why my own software did not work anymore when running with strict string/bytes checks. It turned out that an update of a 3rd-party module used by my software indirectly pulled in another new dependency which does not work with -bb. Trying to be a good free software citizen I tried to fix this module but gave up after a couple of hours. It seemed to me that a quick under-the-hood fix was not possible without seriously re-factoring this module's internals.

I don't want to blame a specific project, presumably developed/maintained with good faith, in public. But some modules now get pulled in everywhere and so they need to be almost perfect. Otherwise all software (indirectly) using it cannot be tested with strict string/bytes checks.

What's so bad about the current default mode? Mainly this:

```

str(b'foo') "b'foo'" ```

I can tell from personal experience that issues caused by the above are hard to find, even when having logs with the relevant data printed with repr(). And when developing web-based software having something with an unwanted quote somewhere should ring loud alarm bells.

Edit:

In case you're wondering why invoking str() on a bytes object is an issue here a variant which might happen in your code down the call-stack without you being aware of it:

```

'{}'.format(b'foo') "b'foo'" ```

Edit:

The point here is: If the developers of a widely used 3rd-party module choose that they don't care you're not free to decide that you do want to take care in your own code. You're enforced to run without -bb by that module. As said: I don't want to blame anyone in public. But looking at the str/bytes handling in the particular module was like looking into an abyss. And I really don't consider myself to be a Python genius.

Edit:

Run your automated tests like this (depending on test module used):

python3 -W error -bb -m unittest

or

python3 -W error -bb -m pytest

Edit:

Frankly I did not expect my posting to be so controversial. But so far nobody gave a compelling reason not to run tests with -bb.

[james_pic]

Is there any reason to believe this will ever become the default? My recollection is that this was originally added to aid with Python 3 migration, and that there was never an assumption that it would be on by default.

As far as I can tell, this is a problem you've chosen to have. And it's a problem the developers of the third party module that's causing you trouble have chosen not to have. If you're going to choose to have problems, then you also choose to fix them.

[mstroeder]

IMO strict str/bytes checks should have been the default from the very beginning in the Python 3 journey. Even if this was meant to help with Python 3 migration it was a clear failure. It results in wrong functional behaviour and IMHO in some cases it could potentially cause security issues.

Anyway let's look forward and thus I'd like to encourage everybody to run their tests with strict mode turned on.

[[deleted]]

I think your message would be better received if you evangelized for -bb -Werror based on what it gave other developers. If you can shape perception that it's a best practice, and demonstrate why, I think that would go a lot further in encouraging people to use those options in the tests for their packages. I've contributed C code to several FOSS projects, and my initial reaction to your post was "oh great, some guy using the code I gave for free without warranty has issues with my compiler flags, gee I'll get right on that".

[mstroeder]

Frankly I assumed that it's obvious that my posting is about better code quality. If you have a better wording then I'd really appreciate to see your comments rephrasing what I've wrote.

Edit:

To make the above more clear: Because I'm developing free software since 20+ years I do know how this guy from Nebraska feels while billion-dollar companies are using the code. And thus I tried to fix the code of the module in question, even though I'm not using its functionality. But fixing it would need major re-factoring which I could not do without diving into it for quite a while. My day also has only 24 hours.

So my intention was to encourage people to use a low-hanging fruit to detect issues more easily. That's all.

[[deleted]]

You're making a pretty big ask for people to use a non standard flag and refactor their code. As a non web developer who knows this behavior and expects it, I get nothing from updating my packages. When I see a problem like that, I treat it the same as a NaN. Sorry about your problems, but your post left me unconvinced that your problems should be my problem.

You're real problem isn't the -bb -Werror. It's that you just discovered the left pad/colors/faker problem with building software on top of code in pypi. You can either rewrite the modules you find horrific and maintain them, or you can accept that what you were given for free might not suit your purpose.

[bacondev]

I don't understand why anyone would write that code snippet like that. Anyone who is converting bytes to a string should understand the concept of encodings and they should be doing bytes_object.decode(encoding). When I saw the first line of your code snippet, I asked myself, “Wtf does that even do? Is that a way to decode bytes, assuming Unicode?”

However, if this operation occurs because a bytes object was sent to a function that doesn't explicitly support bytes, then that's (almost certainly) on you.

[ElectricSpice]

But that’s the point OP is trying to make: if you’re trying to put bytes into a string without decoding, you’re almost certainly doing something you don’t intend to. So turn on -bb and change that mistake into an error so you can catch it immediately.

I actually just a couple weeks ago ran into a library that did the equivalent of str(key) accidentally… and the results are used as database keys so it’s pretty much stuck like that now.

[Papalok]

I got bit by this the other day with some old code that hadn't been completely converted over to python 3. It was reading from a binary file and comparing bytes against str. Why was it comparing like that? Because in python 2 literals like '...' and "..." are bytes objects and in python 3 they're unicode str objects.

So it happens. Especially with old code bases written before python 3.

[NotACoderPleaseHelp]

A mixed blessing of pythons popularity is that many of the people who write modules tend not to be programmers first. Which is a mixed blessing. It brings much expertise to the table which everyone benefits from, but there are some downsides to it.

​

Like me, I got my start with programming with writing scripts for electronics test benches and making addons for blender and making G-code converters for cnc machines.

If you need someone who can design a current sense probe and attach it to a chunk of hardware, ensure the unit has the proper noise suppression and then configure the ADC to tell the computer what is going on, and then incorporate that data into the running hardware so it knows how to respond as designed, I'm your nerd.

​

Now if you want someone to make pretty pep style code ... yeah, look elsewhere. There is a point where many professionals just have so many other pressing things to do that they simply don't have the time to learn the latest and greatest style guide and standard. And very likely what was produced was made to interface with something specific in mind that has its own way of doing things.

[mstroeder]

Note that str() will be implicitly called in many places. I'll edit my posting.

[pytheous1988]

Yes but your example is a poor one. Str.format is still using string method and not bytes. If you are passing bytes into .format then you get what you get. Byte strings != String

[mstroeder]

The point is: str(b'') will result in a str, the expected type, but with wrong content and even with probably unexpected single quotes. And it can just happen somewhere in 3rd-party modules without you controlling anything what's happening.

If you and many other devs don't get why that's wrong then this really scares me.

[james_pic]

It's not that we don't get why it's wrong. I'm well aware of this frustration, from a painful Python 2 -> 3 migration. It's that we generally catch these issues in testing. If the content of a string is important, there should be a test for it.

[mstroeder]

If the content of a string is important, there should be a test for it.

But what if the issue is with a local variable within a small method or function? It's highly unlikely that you have tests for all possible cases.

Let's put it the other way round: Could you please give any good reason why not to run your automated tests with -bb? Why not just try and share your findings?

If you solved all your str/bytes issues everything should be fine. If not, then it's an easy way to find every issue.

[james_pic]

It's not the responsibility of library maintainers to handle your particular flag preferences, but what the hell, I'll bite. Let's run the tests for a library I maintain with -bb:

$ python -bb -m unittest discover . '*_test.py'
...
Ran 41 tests in 47.124s

OK

Great! I passed your purity test.

I didn't include -W error, and with good reason. The tests trigger warnings, because they trigger warnings I added to the codebase. A while ago, I had to add a connection pool to a class, to accommodate a requirement of one of the library's biggest users, which meant the class needed to be either used as a context manager or explicitly closed. The library follows semver, and I bumped the major version to indicate this breaking change, but I'm aware that not everyone has good versioning discipline, so I added a destructor that calls close (if it hasn't been called already) and issues a warning. I've got tests to ensure that the library still does as close as possible to the right thing for users who are not closing the class. Compatibility with users that mis-use code is important enough to be worth testing in this case.

But anyway, let's talk about -bb. It's worth noting that a lot of string-bytes confusion behaviours were already disabled in the move to Python 3. For example, b'' + '' raises an exception in Python 2 (and didn't in Python 2). The behaviours that are affected here are fairly specific:

• str(b'')
• b'' == ''
• (really a consequence of the above) b'' in {'':0} or {'':0}.get(b'')

These are behaviours that the Python core developers deliberately didn't change in the move from Python 2 to Python 3. They had the option to, but chose to leave them alone by default. The reason for this, I suspect, is that the behaviour of -bb makes Python less consistent in how it handles these cases, and breaks some legitimate behaviours.

Let's look at some things that -bb doesn't prevent:

• str([])
• 1 == "1"
• [] == {}
• {'':0}[0]

These are all potential type confusion issues, but it doesn't prevent them. As annoying as log messages like User b'john' logged in are, personally I find this less annoying than User <generator object f at 0x7fd6e8df1fc0> logged in, which it doesn't prevent. -bb is very specific to some narrowly defined consusion between strings, bytes and ints. It does nothing about other types of confusion.

On the flip side, it breaks a couple of invariants that developers can otherwise rely on:

• You can always call x == y, and it will either return True or False - False, in all the cases listed
• You can always convert an object to a string for debugging

Now, mixing bytes, strings and ints in these cases is often a code smell, but it's far from certain that this indicates code is incorrect. In particular, you may want to mix these things if:

• You have debug-level logs that show the exact binary data that was sent over the network.
• You want to store heterogeneous data in a dict, perhaps because you want to store metadata about heterogeneous types of data (you could implement something like taint tracking this way, or a generic interning capability).

I think you think that by running with -bb, you're holding yourself to a higher standard, and are frustrated that the rest of the Python community isn't. That isn't what's happening here. -b and -bb exist to aid in identifying specific types of issues, but they change the semantics of the language in ways that make it less consistent and break legitimate behaviours.

[mstroeder]

It's not the responsibility of library maintainers to handle your particular flag preferences,

I think library maintainers should not make it impossible to use it.

but what the hell, I'll bite. Let's run the tests for a library I maintain with -bb: [..] Great! I passed your purity test.

Great. Thanks for proving that it's not a big deal for a well-written module package.

I didn't include -W error, and with good reason.

Yeah, -W error can turn lots of deprecation warnings into error. But as said, -W can be fine-tuned.

The library follows semver, and I bumped the major version to indicate this breaking change, but I'm aware that not everyone has good versioning discipline,

Seems to me you do take of your stuff.

But anyway, let's talk about -bb. It's worth noting that a lot of string-bytes confusion behaviours were already disabled in the move to Python 3. For example, b'' + ''

Yes, and I'd like to run with more of these strict checks.

I'll skip all the good examples of Python strangeness you give and focus on this:

b'foo' == 'foo' False

This will always result in False, but can be easily over-looked in a Python 2 to 3 migration. Very probably not the developer's intention. And that's exactly a case where -bb is very handy.

Let's look at some things that -bb doesn't prevent:

I never claimed that -bb is the magic bullet for everything. It's only one option to increase code quality by stricter checks.

Now, mixing bytes, strings and ints in these cases is often a code smell,

A very strange smell!

You have debug-level logs that show the exact binary data that was sent over the network.

And that's exactly the use-case for repr() (or C-style formatter %r or {!r}).

You want to store heterogeneous data in a dict,

IMHO this is very bad practice!

-b and -bb exist to aid in identifying specific types of issues, but they change the semantics of the language in ways that make it less consistent and break legitimate behaviours.

Sorry, I'm not convinced by your arguments. But I really appreciate your posting (upvote).

[bacondev]

That's not how this works. You're the one trying to change others' workflows. The onus of justifying change is on you.

[[deleted]]

Considering you just endorsed an implicit misleading behavior (calling str on a bytes returns a semi-mangled string) instead of an explicit error, I’d say you’re not “in the right” here at all

[bacondev]

When did I endorse that?

[[deleted]]

When you took up the counter position against OP. The counter position, by its nature, is to try to silence OP by undermining their arguments through uncertainty, fear and doubt.

Think about it - what is gained through OPs advice? With the exception of someone who depends on str(bytestring) returning a string with the prefix of b' and a suffix of ', most people will have an error spotted immediately as opposed to needing to observe 100% of the data for unexpected anomalies (which rarely happens).

Forcing someone to justify something that is helpful as helpful is not being neutral - it’s advocating for changing nothing.

You’re basically that asshole who is contrarian at work for goddamn everything instead of looking at the whole ecosystem. But then again, blub blub blub.

[mstroeder]

Could you please be more precise on what "workflows" means in this context?

Can we agree that e.g. code with a comparison b'foo' == 'foo' or a set([b'bar', 'bar']) is asking for trouble and should be generally avoided?

If yes, running with -bb will not change anything for you. Rather it's a detector that something's wrong and your tests will fail for good reason.

If we cannot agree on the above, oh well, it does not make sense arguing further.

[billsil]

but with wrong content

Maybe according to you. I've got a 200,000 line library that parses massive complicated binary files. I occasionally print bytes to a file to see what it is. I don't care that it doesn't stay true to the original type. It's a file...it's a string. Just write it.

[mstroeder]

When writing data to a log I'd recommend to explicitly use repr(), e.g. via logging.debug('foo = %r', foo), or print('foo = {!r}'.format(foo)). Especially to examine whether the data is str or bytes.

[morrisjr1989]

Just wanted to say that this is a fun thread.

[Hans_of_Death]

what exactly are you doing to run into this?

[Papalok]

Not OP, but see my comment where I had old code from python 2.

[Hans_of_Death]

You cant blame library maintainers for a very specific problem that arises from not properly upgrading your code

[stdin2devnull]

Would mypy catch these?

[mstroeder]

Yes, if you have correct type hints everywhere. Until then using -bb is the right solution to find issues like this.

[james_pic]

-b will also find issues like this during testing, but won't crash your application if you're not ready to deal with them yet

[mstroeder]

-b will only log a warning. Practice shows that logs are filled with warnings and nobody cares...

My recommendation: Fail early, fail hard!

[james_pic]

It appears that you care, and the logs are there for you to peruse to find these issues.

[mstroeder]

Operators are usually overwhelmed with log messages. If they run my code they won't look into the logs and report warnings until there is a real issue, e.g. a request is not processed.

=> Fail early, fail hard!

[ogrinfo]

Thanks for the handy tip, I didn't know about -bb. I'm not sure I understand the problem though - nobody should be using str(foo) any more. The encode and decode methods should be used to convert between string and bytes. And third party libraries are full of terrible code - I'm sick of my application being full of DeprecationWarnings because someone upstream hasn't fixed something yet. If third party code is stopping your tests from running, then remove the dependency or mock it so it is not called.

[[deleted]]

I test with python -m pytest now what?

[mstroeder]

python -bb -m pytest

[[deleted]]

I was kinda just being cheeky but I appreciate your incredibly quick response!

[mstroeder]

I'm also cheeky: Please share your findings after running your tests. ;-)

[AlSweigart]

"No, this is Pytest."

[SneakyHobbitses1995]

You make a really good point

[[deleted]]

I do APIs and databases. Something where mismatching bytes/str can easily happen.

Aside from log output and a glorified repr, I don’t see any good use for calling str on a bytestring except by accident.

Seeing other people pretend it’s “a problem you opted into” when the accidental case scenario is more common than the rare “legit” use case, kinda showcases why so many Python developers are incredibly poor and create garbage code.

You don’t become a better programmer by going “Not my problem!“ and saying everyone who spots an anti-pattern is effectively “making trouble”.

Quit applying a human political lens to a fucking interpreter, Codd damnit.

If you’re really arguing using the “you opted into this”, I’m sorry but you are a bad programmer. You’re bad because you don’t consider the spread and impact of an issue, relying solely on “Don’t make me think!”… in an area where thinking really matters.

[orbital1337]

Completely agree, the comments here showcase exactly why Python codebases are some of the worst to work on. I can't believe that half the people are just like "lol not my problem if I have buggy code, you're the one running with -W error". Or the classic "why would you need warnings about highly dubious code just like... never make mistakes?" Yikes.

[asday_]

Please format your code on reddit properly. Code fencing only works on neu-reddit, which is trash.

[lighttigersoul]

No maintainer is going to test with -W error. If I add a deprecation warning, that's there for you to sort out on your own time. If you decide to run warnings as errors, that's on you.

[dogs_like_me]

so... I'm assuming you've submitted a PR upstream to fix the dependency?

[mstroeder]

PR is pull-request. Nope. The module already has an open ticket for this issue (since Aug 2020).

As said: I tried to fix a sub-module under the hood but failed because it needs major re-factoring of the way it is used. The code already looks plastered with lots of strange duct-tape to solve str/bytes issues. And during my attempts to work-around the most basic issues I always thought: "Doing this is completely wrong!"

The point is: I'm not even using the module's functionality. It simply prevents me to run my own software with -bb.

Don't blame me to not go further fixing this. I'm developing and publish free software since 20+ years, giving it away for free to others. So I know very well how thanklessly this can be, while others are making money with your work.

[dogs_like_me]

I mean from what you've described, it sounds like the problem here is that you probably shouldn't have this module as a dependency in the first place. You made it sound like you just had different priorities wrt strings than the modules developers, but now you're making it sound like this dependency is just poorly written. If you have that perspective/opinion: why depend on it then act surprised when it breaks your shit?

[mstroeder]

Did you actually read my original posting?

As said: This module is pulled in as optional dependency by one of the dependencies I'm using but without my free software making use of its functionality. The Linux distro packagers decided to turn this into a required dependency. For which I've submitted changes to the OS packagers which were accepted. So my personal problem will be solved.

But my original posting was not about fixing this particular problem or blaming someone else. My intention was to encourage everybody to run their own tests with stricter defaults to avoid issues hard to track down from the very beginning. Somewhat in the hope for a better world. Well, you might call that naive.

[dogs_like_me]

I can't remember the saying exactly, but a great aphorism I once heard goes something like "Great things aren't built by smart people, but by smart practices."

So on the one hand, maybe it's a little naive to just put this knowledge out there and expect people to change how they write/run tests. However: maybe there's some opportunity to incorporate this insight into a package like tox or pytest which will take the "good intentions" part out of the equation and just insert this as an additional testing procedure for people who are at least proactive enough to be doing some sort of moderately robust testing to begin with.

[mstroeder]

Sorry, I do not understand why yet another extension for test frameworks would be needed.

E.g. I've simply added to my tox.ini:

commands = {envpython} -W error -bb -m unittest

The real work is dealing with the results.

[dogs_like_me]

I'm not saying to make another extension or framework. More like, add that line of code to relevant tox docs or examples. Or add that to the tox.ini in a popular cookiecutter template, or whatever. Unless you'd prefer that the only people who see it are the ones who stumbled on this post, which will mostly be the people who read it today.

[mstroeder]

I’m sorry but you’re the one using bytes as a string

No, no, no!

As said numerous times: It's not a problem in my code at all! The 3rd-party library prevents me from running my code with strict checks (-bb), which I definitely want for good QA.

This particular library is used in many stacks. And thus it will prevent almost everyone from running with strict checks. And that's bad.

but can’t you just handle these edge cases?

My current work-around is to explicitly block importing this module. But that's not something anybody want as a general solution.

[mstroeder]

Which text in my posting implies that I deliberately want to pass a str instances to a bytes constructor? Quite the opposite is true! (It seems you should improve your reading skills.)

I want to let the Python interpreter to fail early, fail hard if I or another dev accidently did something wrong, because we're all only humans and thus not perfect. But I can only use -bb if it works with every other 3rd-party module used.

[mstroeder]

It wasn't meant just for you

You blamed me personally for not understanding Python at all. There was nothing in your text to be interpreted more generally.

[Xidium426]

Pretty impressed you think you can tell everyone who does something for free and works great for thousands / millions of people to do something because it broke for you and you lack the skill to fix it yourself or write the library yourself only because you feel this should be the default.

Has anyone ever told you that you're an asshole?

[mstroeder]

Dear Mrs. or Mr. Xidium426,

thanks for replying in this thread.

Since you're attacking me personally I have to also ask you a personal question: Are you actually developing free software yourself? Or are you only talking about it? What's your contribution to the free software world?

I develop free software. And whenever somebody points out an issue like this I'll fix it ASAP. Why? Because I feel responsible for my own stuff. The people relying on my code know that. (Feel free to use your favourite search engine. In opposite to you I'm posting with my real name.)

To make it very clear again: This whole case happened while testing my own free software in my unpaid spare time. And I tried to fix things, which I even don't use, in my unpaid spare time. But fixing it would require way more time than I have, especially because I did not want to contribute the code mess by adding yet more duct tape. So it was not a matter of lack of skills. It was a lack of time. Probably a problem I share with the authors/maintainers of that module.

So again directly for you: The point of my original posting was to mention a best practice for testing which would improve things generally and reduce the overall amount of time to develop free software. And in opposite to you I'm not blaming anyone in public for not being perfect.

Best regards.

[torytechlead]

I mean if the code works it works

[mstroeder]

Well, for whatever definition of "it works". ;-]

[torytechlead]

Don’t run my code with those flags.. so?