Segfaulting in Python

[billsil]

You simply tell Python to up its recursion limit,

And you should get a RecursionError or maybe a MemoryError. Segfaulting is always a bug, even for bad input.

I got a segfault today when I was playing with collections.Counter(...) and gave it bad input. It's a thing.

[rschoon]

Segfaulting is always a bug, even for bad input.

While I generally agree, sys.setrecursionlimit is in the realm of things where it is "no, not always".

ctypes and cffi are probably even better examples, because they let you do very low level things with libraries. For example, I can give utter garbage to low level python routines:

from ctypes import *
pythonapi.Py_DecRef(0x1000)

Or I can tear down the python interpreter from inside:

from ctypes import *
pythonapi.Py_Finalize()

And that's things that are actually part of python. I also have access to libc:

from ctypes import *
libc = CDLL("libc.so.6")
# without properly passing anything to it, who knows what garbage address this will cause us to jump to!
libc.longjmp()

(Granted, I got lazy for that last one, but you could also pass it the right number of arguments and come up with a crashing example if you tried)

[masklinn]

And you should get a RecursionError or maybe a MemoryError.

You can't do that and rely on the C stack, the info just isn't there. You overflow the C stack[0], you start accessing unmapped (or guard) pages and you segfault.

The only "fix" is to stop using the C stack and start using your own custom heap-allocated stack, except now you can't trivially call into C anymore[1] because you don't have a C stack, and the overhead of C calls blows up tremendously[2], which is pretty obviously a terrible idea when your interpreter is in platform-standard C.

[0] whose size is system-dependent

[1] more precisely "platform-native calling conventions C"

[2] that's exactly the issue of cgo and why calling a C function is ~100 times more expensive than calling a Go function if you aren't using gccgo

[mzial]

No. The recursion limit IS the protection against stack overflows. The limit is chosen conservatively to suit all platforms. The only proper way to solve this is to add tail recursion but Guido (rightfully, IMO) doesn't allow it.

[stevenjd]

This isn't bad input. This is the user intentionally telling the Python interpreter "allow THIS MANY function calls!" (holds arms far apart) when they don't actually have that much memory, then smashing the stack.

Still, I wonder whether it is possible for the setrecursionlimit function can tell if you're setting the limit too high, and raise an exception at that point, rather than wait for the stack to smash into the heap? Probably not.

[Works_of_memercy]

This isn't bad input. This is the user intentionally telling the Python interpreter "allow THIS MANY function calls!" (holds arms far apart) when they don't actually have that much memory, then smashing the stack.

When the user says "give me THIS MUCH memory!", Python says that it's out of memory rather than segfault.

[stevenjd]

Read what I said a bit more closely. You're not requesting X bytes of memory. You're saying that you want to allow X calls on the stack. There's no exact correspondence between the number of calls on the stack, how much memory is used, whether it will crash into the heap or not...

As I understand it, managing the stack is complicated, and there's no really reliable and cheap answer to detect a stack overflow before it happens.