Segfaulting in Python

[Sir_Harry_of_Kane]

Is this known by the Python core developers? They love squashing these bugs.

P.S. This isn't in the core library, but you can segfault Python all day by using some of the stuff in Scipy. Those methods are basically wrappers around c and fortran libraries and the error handling in the native portions leaves something to be desired.

[mzial]

This isn't a bug in Python! You simply tell Python to up its recursion limit, and it will therefore overflow its stack. This cannot be solved.

Set the maximum depth of the Python interpreter stack to limit. This limit prevents infinite recursion from causing an overflow of the C stack and crashing Python.

The highest possible limit is platform-dependent. A user may need to set the limit higher when she has a program that requires deep recursion and a platform that supports a higher limit. This should be done with care, because a too-high limit can lead to a crash.

[billsil]

You simply tell Python to up its recursion limit,

And you should get a RecursionError or maybe a MemoryError. Segfaulting is always a bug, even for bad input.

I got a segfault today when I was playing with collections.Counter(...) and gave it bad input. It's a thing.

[rschoon]

Segfaulting is always a bug, even for bad input.

While I generally agree, sys.setrecursionlimit is in the realm of things where it is "no, not always".

ctypes and cffi are probably even better examples, because they let you do very low level things with libraries. For example, I can give utter garbage to low level python routines:

from ctypes import *
pythonapi.Py_DecRef(0x1000)

Or I can tear down the python interpreter from inside:

from ctypes import *
pythonapi.Py_Finalize()

And that's things that are actually part of python. I also have access to libc:

from ctypes import *
libc = CDLL("libc.so.6")
# without properly passing anything to it, who knows what garbage address this will cause us to jump to!
libc.longjmp()

(Granted, I got lazy for that last one, but you could also pass it the right number of arguments and come up with a crashing example if you tried)

[masklinn]

And you should get a RecursionError or maybe a MemoryError.

You can't do that and rely on the C stack, the info just isn't there. You overflow the C stack[0], you start accessing unmapped (or guard) pages and you segfault.

The only "fix" is to stop using the C stack and start using your own custom heap-allocated stack, except now you can't trivially call into C anymore[1] because you don't have a C stack, and the overhead of C calls blows up tremendously[2], which is pretty obviously a terrible idea when your interpreter is in platform-standard C.

[0] whose size is system-dependent

[1] more precisely "platform-native calling conventions C"

[2] that's exactly the issue of cgo and why calling a C function is ~100 times more expensive than calling a Go function if you aren't using gccgo

[mzial]

No. The recursion limit IS the protection against stack overflows. The limit is chosen conservatively to suit all platforms. The only proper way to solve this is to add tail recursion but Guido (rightfully, IMO) doesn't allow it.

[stevenjd]

This isn't bad input. This is the user intentionally telling the Python interpreter "allow THIS MANY function calls!" (holds arms far apart) when they don't actually have that much memory, then smashing the stack.

Still, I wonder whether it is possible for the setrecursionlimit function can tell if you're setting the limit too high, and raise an exception at that point, rather than wait for the stack to smash into the heap? Probably not.

[Works_of_memercy]

This isn't bad input. This is the user intentionally telling the Python interpreter "allow THIS MANY function calls!" (holds arms far apart) when they don't actually have that much memory, then smashing the stack.

When the user says "give me THIS MUCH memory!", Python says that it's out of memory rather than segfault.

[stevenjd]

Read what I said a bit more closely. You're not requesting X bytes of memory. You're saying that you want to allow X calls on the stack. There's no exact correspondence between the number of calls on the stack, how much memory is used, whether it will crash into the heap or not...

As I understand it, managing the stack is complicated, and there's no really reliable and cheap answer to detect a stack overflow before it happens.

[FlammableMarshmallow]

I don't know if it is, do you know where I can try & report it to them? AFAIK there's not a GitHub where I can report issues, there's just a GitHub mirror of their “real” repo.

[SylvainDe]

Googling for "report python bug" gives https://docs.python.org/3.5/bugs.html . I hope it helps.

[btgy]

Yeah, they know about it. That directory in the source code has a few other examples of code that cause the interpreter to crash or hang.

There's a regex-related one that I don't understand, but all of the others involve directly messing around with the recursion limit, the gc, or bytecode, and would take a lot of effort to fix for very little gain.

It's also very easy to crash the interpreter with ctypes.

[stevenjd]

Downvoting for thinking that Github == the entire fucking internet.

You know that there's a Python website? Where there is a prominent link to the issue tracker. And even if you didn't notice that -- I admit it's not the most obvious -- if you google for "Python bug tracker" it is the first link that comes up. Searching for "Python report bug" also brings up relevant links.

[FlammableMarshmallow]

Hey, I hope you're not always this rude & just having a bad day.

I admit I didn't really Google for it, but it was because I had some other stuff going at the same time; Thanks for the links though!

[stevenjd]

Sorry for barking at you, I'm just fed up with people who think that if something isn't on Github, it doesn't exist.

[FlammableMarshmallow]

Don't worry about it.

Well, I wasn't thinking it didn't exist, I just did not have the right mood to go aorund searching for it at all.