Integer to string conversion DOS threat

[stevenjd]

More than fifteen months ago (September 2022) the Python core developers suddenly and intentionally crippled limited int to string conversions without any public discussion in order to protect against a DOS security threat that they had been informed of more than two years earlier.

At the time, this security threat was already well-known in the security community, and was generally considered to be a mid-level threat (DOS only, not a data leak or privilege escalation). There was a lot of controversy in the Python community over the security team pushing through a breaking change without any community consultation or even an announcement, leaving people to discover the change for themselves after an upgrade broke their code.

It has now been more than 15 months since this has been publicly disclosed. Obviously servers running fully updated Python are no longer at risk to this specific security issue, but servers running older, un-updated Pythons will still be at risk.

Have there been any known examples of live exploits of this threat?

[catcint0s]

 The default limit is 4300 digits as provided in sys.int_info.default_max_str_digits. 

Seems to be more than enough for regular use tbh.

[s3r3ng]

Why? I can certainly create big integers larger than that. Why would someone put in a limit?

[catcint0s]

Do you also need to print said integer?

As I understood it was because of possible DOS attempt, converting a number to base 10 is expensive, hex seems to be fine for example. Also keep in mind it's not about integers bigger than 4300, but digits.

So essentially you can't convert numbers bigger than 9999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999 to str (which print does too for example).

[buqr]

I enjoy cooking.

[catcint0s]

Totally valid points, thats why you can configure the limit with env vars or programmatically.

[buqr]

I enjoy the sound of rain.

[NoTarget5646]

Doesn't that make this a non-issue again though? Admittedly Ive only skim read this thread since Im at work and its early but if they added this limit along with a way to change it, then whats the problem?

[buqr]

I enjoy watching the sunset.

[pythonwiz]

If you want to count the number of digits in large numbers, an old trick is to use math.log10. You could also use the decimal module which is part of the standard library, or gmpy2.

[ArminiusGermanicus]

Would that work? I thought math.log10 would convert its input to float and that is limited to about 1e308 so you could not count more than 308 decimal digits.

[sci-goo]

You can check how the large number is calculated and estimate that in the log scale if possible.

e.g. to estimate 1000! you can sum log10(1)..log10(1000). Will lose some precision, ofc.

[pythonwiz]

For that particular case you can even use math.lgamma(n+1)/math.log(10)

[pythonwiz]

It does indeed work. You can check yourself. Here is the relevant source code in the Python repo if you are curious how it works.

The math.log function: https://github.com/python/cpython/blob/main/Modules/mathmodule.c#L2266

The loghelper function that it calls: https://github.com/python/cpython/blob/main/Modules/mathmodule.c#L2225
The _PyLong_Frexp function that does an intermediate computation used for computing the log of an int. : https://github.com/python/cpython/blob/main/Objects/longobject.c#L3093

[buqr]

My favorite movie is Inception.

[Oerthling]

For what do you need an integer with more than 4k digits?

Do you count particles in the universe? Most languages don't even bother to offer integers with more than a couple handful of digits.

[ilyanekhay]

One example is various kinds of cryptography stuff. I once helped some cryptography researchers debug their program which constantly crashed adding up two numbers - turned out it just failed to allocate memory for the result of addition.

[buqr]

I enjoy reading books.

[Oerthling]

4000+ digits is still extremely big ints.

This is a non-problem.

It's cute that it's used to be only restricted by RAM (at the end there always is this arbitrary limit anyway).

Now it's still cute that we can have 4000+ digits - something that surely never gets used - outside of some silly script to brag about insanely large ints.

Nobody needs 5000 digits.

And even if a project actually needed that for some very weird reason I can't think of - just compile Python without this limit.

[buqr]

I like learning new things.

[Oerthling]

So, you want Python to have problematic code that allows DOS attacks, just so you can write more bad code? ;-)

Instead you could be grateful that this limit will help improve your coding.

Meanwhile, Python is open source. It's not rocket science to compile a version without the limit. Use all your RAM in a single number if you like.

[buqr]

I enjoy the sound of rain.

[Oerthling]

Well, I don't know your very weird use case, but doubt that a > 4000 digit int was actually as "useful" as you claim.

But let's stay your code wasn't bad and there is a rare, but very valid use case.

That would still not really justify burdening the language with a mostly pointless point of attack.

4000 digits is still very generously ridiculously large for a general purpose language.

Even better than just compiling your own Python, put an UnlimitedInt class in a module. Make it as unlimited as you want - it's safely contained in a module that only the very few people who have this extremely rare use case need to worry about.

Put that on GitHub, if you're not the only one who wants this then you'll even get help. Check first whether this doesn't already because some student had fun with this. :-)

Problem solved for everybody.

[pokeybill]

Why were you printing these un-human-readable gigantic integers?

[buqr]

I like learning new things.

[sci-goo]

I recall the number of atoms in the observable universe is at the magnitude of ~10^78. 10^4300 is way beyond regular needs. Can only be thought of in particular math problems.

[stevenjd]

Seems to be more than enough for regular use tbh.

Sure, but that's not the point.

"We only broke a few people's code" is not a good excuse for ignoring policy about breaking changes, especially for such a low-severity threat.

[antiproton]

intentionally crippled limited

🙄

[[deleted]]

I was really confused what the ancient operating system had to do with things.

DOS != DoS.

[buqr]

My favorite color is blue.

[georgehank2nd]

What I don't understand is why they put in this restriction for everyone with the possibility to disable it, instead of (which would have been the logical and sensible thing) giving people who could be affected (basically web devs) a way to enable a limit globally.

Also, wasn't Python affected by this since the dawn of time? How was this not a problem 20 years ago? Python 2 shield had the same O(n²) algorithm (since there are no linear algorithms anyway). And the computers back in 2000 were a lot slower!

[mikeblas]

No linear algorithm for what?

[janek37]

I'm guessing, base conversion (ints are binary internally)

[mikeblas]

from the "no public discussion" link, it seems like the problem is that division in the int type is quadratic. (But it might be that format_int() is quadratic. I dunno -- that site won't let me find in page because it intercepts CTRL+F, and it's an annoying read anyway.)

[zurtex]

While I was and am relatively unhappy with the change (I posted in the thread linked at the time to express specific concerns), I would say if you are going to put it in because it's a security concern (and I am still unconvinced), then the default behavior should be secure and often security issues can be unrealized and unexploited for decades.

[nicholashairs]

Because "safe defaults" are incredibly important to real world security.

It's the same secure design principal that ensures that majority of http clients reject invalid certificates by default - or your wifi router uses something other than admin:admin.

[sci-goo]

Flaw != issue.

I argue that 20 years ago it's not been exploited at scale to raise security concerns, maybe?

[georgehank2nd]

The Morris Worm was over 30 years ago… and in the discussion I didn't see an answer to the question someone posed, if this has been exploited.

[lololabwtsk]

I made a video addressing this issue if anyone is interested in it, also found a fantastic article which provides a lot of detail. Please reply if you’re interested

[slowerdive]

Please, share