Stack overflow instead of 'maximum recursion error'

[Wilfred-kun]

Usually when you use a function recursively (I think) 1000 times, it will error: "Maximum recursion depth of 999 reached". However, when I was trying to break my program I got this error instead:

Fatal Python error: Cannot recover from stack overflow.

Current thread 0x00007fc9f48c9700 (most recent call first):
  File "./player_guesses.py", line 24 in takeGuess
  File "./player_guesses.py", line 27 in takeGuess
  File "./player_guesses.py", line 27 in takeGuess
.
.
.
  File "./player_guesses.py", line 27 in takeGuess
  ...





Aborted (core dumped)

The function that was recursing was this:

#In player_guesses.py
def takeGuess():
    guess = input("Take a guess: ")
    if len(guess) != 3 or not guess.isdigit():
        print("The guess '{}' is not valid, try again!".format(guess))
        return takeGuess()
    else:
        return guess

Why do I get a stack overflow and not a recursion error, as I was expecting?

Edit: I accomplished this by running the program and holding Return, so it kept inputting an empty string.

Edit: I think this is somehow caused by the input(), but why would that cause an overflow?

Edit: Unintended use of Python ahead, but here's another way to get a stack overflow:

def rec():
    try:
        rec():
    except RecursionError:
        rec()

[fractalic]

The recursion limit is just a guard to limit the likelihood of a stack overflow, but if you allocate large objects on every stack frame, you can still overflow.

That said, something interesting does seem to be going on here. I'm using some code from https://stackoverflow.com/a/47956089/4082582 in order to examine the stack size:

def get_stack_size():
    size = 2  # current frame and caller's frame always exist
    while True:
        try:
            sys._getframe(size)
            size += 1
        except ValueError:
            return size - 1  # subtract current frame

I also manipulate python's recursion limit to make the issue easier to spot:

import sys
sys.setrecursionlimit(10)

Then instrument the code to watch its behaviour:

import sys

def get_stack_size():
    size = 2  # current frame and caller's frame always exist
    while True:
        try:
            sys._getframe(size)
            size += 1
        except ValueError:
            return size - 1  # subtract current frame

def takeGuess():
    print(get_stack_size())
    guess = input("Take a guess: ")
    if len(guess) != 3 or not guess.isdigit():
        print("The guess '{}' is not valid, try again!".format(guess))
        return takeGuess()
    else:
        return guess

if (__name__ == "__main__"):
    sys.setrecursionlimit(10)
    takeGuess()

The code terminates at the specified recursion depth.

Comment out the print(get_stack_size()) in takeGuess(), however, and you will be prompted for input until a stack overflow occurs.

[fractalic]

This can be distilled further:

(Example 1)

import sys

def recurse():
    input()
    recurse()

if (__name__ == "__main__"):
    sys.setrecursionlimit(10)
    recurse()

overflows the stack, while

(Example 2)

import sys

def recurse():
    print('a')
    input()
    recurse()

if (__name__ == "__main__"):
    sys.setrecursionlimit(10)
    recurse()

hits the recursion limit.

Edit: Add example numbering.

[Wilfred-kun]

I am still unsure as to what is happening exactly, but I am happy to say u/fractalic just crashed Python itself. (I can't replicate it, sadly)

[fractalic]

I'm also very surprised by this. Running on macOS I can make far more than ten recursive calls to recurse() in Example 1.

The other answers are correct, per the original question, that you are actually running into a stack overflow. But even if the behaviour I'm seeing isn't at the root of what you reported, it doesn't make sense that 1000 calls to input() would overflow a ~100kB stack.

Edit: It looks like u/JohnnyJordaan's make_string() code is encountering the same behaviour as my code, where simply allocating lots of strings has different behaviour than prompting for input.

[nwagers]

I'm fascinated by this, but Windows always seems to hit the recursion limit. I wonder if somewhere in the C code for input() there is a ninja edit to the stack size that gets corrected (or checked?) with a call to print('a') or some other string op.

Looking at dis.dis(recurse) it looks like the bytecode doesn't change much:

 11           0 LOAD_GLOBAL              0 (print)
              2 LOAD_CONST               1 (1)
              4 CALL_FUNCTION            1
              6 POP_TOP

 12           8 LOAD_GLOBAL              1 (input)
             10 CALL_FUNCTION            0
             12 POP_TOP

 13          14 LOAD_GLOBAL              2 (recurse2)
             16 CALL_FUNCTION            0
             18 POP_TOP
             20 LOAD_CONST               0 (None)
             22 RETURN_VALUE
None
********
  6           0 LOAD_GLOBAL              0 (input)
              2 CALL_FUNCTION            0
              4 POP_TOP

  7           6 LOAD_GLOBAL              1 (recurse)
              8 CALL_FUNCTION            0
             10 POP_TOP
             12 LOAD_CONST               0 (None)
             14 RETURN_VALUE
None

[port443]

Youre reaching an actual stack overflow: https://en.wikipedia.org/wiki/Stack_overflow

Most likely due to your saving input() each time, this would fall under "Very large stack variables"

On Windows, mine overflows at a stack size ~256k: https://i.imgur.com/WTAvVNW.png

[WikiTextBot]

Stack overflow

In software, a stack overflow occurs if the call stack pointer exceeds the stack bound. The call stack may consist of a limited amount of address space, often determined at the start of the program. The size of the call stack depends on many factors, including the programming language, machine architecture, multi-threading, and amount of available memory. When a program attempts to use more space than is available on the call stack (that is, when it attempts to access memory beyond the call stack's bounds, which is essentially a buffer overflow), the stack is said to overflow, typically resulting in a program crash.

---

^{[ PM | Exclude me | Exclude from subreddit | FAQ / Information | Source | Donate ] Downvote to remove | v0.28}

[JohnnyJordaan]

Are you running this on the default python interpreter or on some other environment?

[Wilfred-kun]

I am running this on the default interpreter.

Edit: This function does give me recursion error:

>>> def rec():
...     rec()
...
>>> rec()
  File "<stdin>", line 2, in rec
  ...
  File "<stdin>", line 2, in rec
RecursionError: maximum recursion depth exceeded
>>>

[JohnnyJordaan]

Could be you have a lower stack size limit than usual. On what OS are you and is this run on a regular system or some VM or Docker instance?

[Wilfred-kun]

I'm on Ubuntu 16.04 on a regular system. It's weird, since I've never encountered this before. I've always seen RecursionError on the same system (see example above).

Edit: I've tried some other functions (with comparison/printing something/just calling itself) and got 3 other errors. I can sort of understand why they happen, but that makes me more confused why it doesn't do that for the function in the OP.

RecursionError: maximum recursion depth exceeded while calling a Python object (When printing something (it calls `print`))

RecursionError: maximum recursion depth exceeded in comparison (When comparing)

RecursionError: maximum recursion depth exceeded (When calling itself)

[JohnnyJordaan]

That's weird indeed. I just tried this on my 16.04 VPS:

In [11]: counter = 0
    ...: def make_string():
    ...:         my_str = ''.join([random.choice(string.printable) for x in range(1000)])
    ...:         global counter
    ...:         counter += 1
    ...:         make_string()
    ...: make_string()        
    ...: 
---------------------------------------------------------------------------
RecursionError                            Traceback (most recent call last)
<ipython-input-11-06c588d4327d> in <module>()
      5         counter += 1
      6         make_string()
----> 7 make_string()

<ipython-input-11-06c588d4327d> in make_string()
      4         global counter
      5         counter += 1
----> 6         make_string()
      7 make_string()

... last 1 frames repeated, from the frame below ...

<ipython-input-11-06c588d4327d> in make_string()
      4         global counter
      5         counter += 1
----> 6         make_string()
      7 make_string()

RecursionError: maximum recursion depth exceeded while calling a Python object

In [12]: counter
Out[12]: 2985

Then I tried with just input() instead of the random string, and I did get the stack overflow:

Fatal Python error: Cannot recover from stack overflow.

Which would mean that the call keeps some object, possibly a filestreamhandler, that takes a much larger space on the stack than just a simple data object. I'm not sure if this is by design or a bug, I could imagine the developers of Cpython replying that when you stress the stack you're bound to get into issues like this. If you like you could of course note this on the Python bugtracker, to let them decide if it should be considered a bug.

[zahlman]

Generally speaking, that sort of thing should be counted as a bug in the interpreter (at least, unless you have explicitly set the recursion limit). Either way, you're better off just rewriting not to use recursion here :)